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ARI  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&D  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  irnplementatio  at  the  time  of  publication  ate  presented  in  the  last  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  official  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


The  research  reported  here  is  part  of  a  broade*-  program  on  training 
for  combat  effectiveness  being  conducted  by  the  US  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences.  The  availability  of  knowledge 
regarding  current  combat  readiness  of  the  troops  and  the  existing  skill 
deficiencies  is  critical  to  resource  allocation  and  training  management  in 
units . 

The  ARI  Field  Unit  at  Fort  Banning,  Georgia  is  investigating  the 
feasibility  and  utility  of  a  partial  answer  to  this  need:  the  use  of  unit 
training  devices  for  evaluation/qualification  purposes  as  well  as  for 
training.  This  report  describes  findings  regarding  three  aspects  of 
feasibility:  acceptability  to  end-users,  technical  feasibility,  and 
financial  feasibility.  As  a  part  of  the  technical  feasibility  question  and 
also  so  as  to  address  the  utility  question,  the  application  of  a  Cost  and 
Information  Effectiveness  Analysis  method  to  alternative  training  devices 
and  other  evaluation  methods  for  M16A1  rifle  skills  is  described  and  the 
results  are  presented. 

With  regard  to  the  three  aspects  of  feasibility,  the  use  of  training 
devices  to  assess  combat  readiness  appears  to  be  a  satisfactory  technique. 
With  regard  to  utility,  the  use  of  a  training  device  for  these  purposes  for 
M16A1  rifle  skills  is  viewed  as  valuable  by  military  personnel. 


f JOSEPH  ZEIDNER 
>Wthnical  Director 


TRAINING  DEVICE  OPERATIONAL  READINESS  ASSESSMENT  CAPABILITY  (DORAC) : 
FEASIBILITY  AND  UTILITY 

BRIEF 


RequI rement : 


Up  to  date  information  on  the  combat  readiness  and  skills  deficiencies 
of  troops  is  needed  to  report  personnel  readiness  levels  and  to  manage  troop 
training.  Meeting  this  need  can  be  made  difficult  by  frequent  personnel 
turnover;  infrequent  testing;  and  unknowns  regarding  skill  decay  rates. 

A  partial  solution  to  this  problem  may  reside  in  unit  training  devices 
that  are  currently  fielded  and/or  under  development.  If  a  device  is 
designed  ani  used  to  satisfy  training  requirements,  it  may,  in  many 
cases,  also  be  designed  and  used  on  a  regular  basis  (e.g.,  quarterly) 
to  provide  information  on  personnel  qualification  status  and  skill 
deficiencies  requiring  training.  Requirements  in  developing  this  solution 
include  an  examination  of  the  operational  unit  environments  within  which 
it  would  be  employed,  and  evaluation  of  methods  needed  to  implement  the 
solution  in  the  context  of  selected  training  devices. 


Procedure : 

The  investigation  consisted  of  two  parts:  (1)  a  field  survey  of  US  Army 
Forces  Command  (FORSCOM)  units  located  at  three  Army  posts  and  (2)  a  trial 
application  of  a  newly  developed  multiattribute  utility  measurement  (MAUM) 

Cost  and  Information  Effectiveness  Analysis  (CIEA)  method  to  evaluation 
alternatives  for  M16A1  rifle  marksmanship  skills.  The  field  survey  consisted 
of  interviews  with  and  administration  of  questionnaires  to  command  and  staff 
personnel  responsible  for  combat  readiness  status  and  status  reporting,  and  for 
training  management.  The  survey  obtained  information  concerning  current  prac¬ 
tices  in  reporting  troop  readiness  and  skill  levels,  deficiencies  in  the 
training  evaluation  and  unit  status  reporting  systems,  perceived  ways  to 
improve  the  systems,  and  DORAC  concept  acceptability  and  perceived  utility. 

The  MAUM  CIEA  method  was  applied  to  combinations  of  M16A1  rifle  marksmanship 
evaluation  alternatives  which  included  two  training  devices  and  the  current 
field  testing  procedure.  Record  Fire.  The  combinations  varied  in  terms  of 
number  of  alternatives  included  (e.g..  Record  Fire  alone  vs.  Record  Fire  and 
one  or  more  of  the  training  devices)  and  the  frequency  of  use.  Personnel 
involved  in  application  of  the  CIEA  method  included  Army  officers  from  the 
FORSCOM  and  US  Army  Training  and  Doctrine  Command  (TRADOC) ,  who  made  judgments 
regarding  the  information  utility  of  each  of  the  alternatives;  and  contractor 
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personnel,  win  compiled  the  cost  data  and  produced  the  final  values  which 
reflected  the  tradeoff  between  the  information  utility  and  cost  figures  for 
each  alternative.  The  MAUM  CIEA  method  is  described  ir  detail  in  a  companion 
report  (Hawley  &  Dowdy,  1981). 


Findings: 

(1)  Use  of  a  DORAC  is  quite  acceptable  to,  and  conridered  desirable  by, 
those  FORSCOM  officer  and  enlisted  personnel  who  have  had  prior  experience 
(e.g.,  in  Air  Defense  trainees  evaluate  team  prof iciency /readiness) .  Those 
who  have  not  are  doubtful  and  need  assurance  regarding  data  validity  and  the 
adequacy  of  device  maintenance. 

(2)  The  needs  expressed  by  FORSCOM  officer  and  enlisted  personnel  for 
more  standardized  evaluation  procedures  and  performance  standards,  and  for  more 
objective  performance  measures,  are  exactly  the  needs  that  implementation  of 
DORAC  would  meet. 

(3)  The  MAUM  CIEA  method  is  usable,  with  contractor  assistance  at  least, 
by  FORSCOM  and  TRADOC  officer  personnel.  Of  particular  importance  is  that 
these  personnel  are  able  to  make  judgments  of  the  information  utility  of  data 
from  a  number  of  evaluation  alternatives  using  MAUM  CIEA  procedures  and  to 

do  this  consistently  in  terms  of  judged  relative  importance  of  information 
about  each  of  the  skill  components  of  rifle  marksmanship. 

(4)  There  is  a  need,  however,  to  simplify  and  expedite  the  application  of 
the  CIEA  method  to  the  extent  possible.  This  need  will  be  explored  in  future 
research. 

(5)  Training  devices,  when  analyzed  carefully  with  respect  to  the  value 
of  having  and  using  a  DORAC  in  operational  units,  can  be  judged  to  be  useful 
for  evaluation  purposes.  The  TRADOC  and  FORSCOM  personnel  applying  the  MAUM 
CIEA  method  to  the  M16AI  DORAC  alternatives  judged  the  combination  of  one  of 
the  training  devices  with  Record  Fire  to  have  a  considerably  higher  information 
utility  than  Record  Fire  alone. 


Utilization  of  Findings: 

Findings  will  be  used  as  inputs  to  further  work  in  developing  the  MAUM 
CIEA  method,  possible  efforts  to  develop  alternative  CIEA  methods,  and  work  to 
develop  a  DORAC  and  vse  procedures  in  selected  training  devices.  The  ultimate 
users  of  the  final  CIEA  methods  will  be  TRADOC  training  developers  who  are 
device  and  system  proponents,  and  device  design  engineers  working  with  the 
materiel  developer  (PM  TRADE).  The  ultimate  users  of  information  obtained  from 
DORAC  usage  of  any  particular  device  will  be  unit  personnel,  ranging  from  the 
trainers  to  the  commanders. 
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SECTION  1 


INTRODUCTION 

Background 


Moat  scenarios  for  a  full-scale  confrontation  between  the  United  States 
and  any  of  its  major  potential  adversaries  indicate  that  the  majority  of 
Army  units  will  have  to  be  prepared  to  fight  immediately  without  the  luxury 
of  a  lengthy  mobilization  period,  such  as  the  first  year  of  World  War  II. 
Studies  of  the  comparative  military  strengths  of  the  United  States  and  the 
Warsaw  Pact  countries  also  indicate  that  U.S.  forces  are  likely  to  be  heav¬ 
ily  outnumbered,  often  by  a  ratio  of  five  to  one  or  more.  To  have  any  hope 
of  success  in  this  "come  as  you  are  and  win  the  first  battle  outnumbered" 
situation,  the  Army  will  have  to  maintain  a  high  level  of  individual  and 
unit  combat  readiness  at  all  times.  Maintaining  a  consistently  high  level 
of  combat  readiness  will  necessitate  frequent  evaluations  of  individual  and 
unit  proficiency,  along  with  a  means  of  quickly  diagnosing  and  remediating 
performance  deficiencies. 

In  simpler  times,  the  assessment  of  individual  and  unit  job  proficiency 
presented  no  special  difficulties.  Recently,  however,  the  complex  nature 
of  many  weapons  systems,  increased  personnel  turbulence,  rapid  skill  decay 
rates,  and  changes  in  training  philosophy  have  led  to  an  emphasis  on  per¬ 
formance-oriented  training  and  criterion-referenced  testing.  Soldiers  are 
required  to  demonstrate  their  individual  or  collective  competency  in  a  "hands 
on"  environment  using  actual  equipment.  This  change  in  training  and  evalua¬ 
tion  methodology  has  increased  the  demand  for  training  and  evaluation  uses 
of  operational  equipment  and  accompanying  support  resources  requirements 
[e.g.,  ammunition,  spare  parts,  POL  (petroleum,  oil,  and  lubricants),  etc.] 
during  a  time  of  inflation  and  budgetary  constraints.  More  frequent  per¬ 
formance-oriented  individual  or  collective  readiness  evaluations  would  thus 
tend  to  complicate  an  already  tight  situation. 

A  proposed  solution  to  the  problem  of  conducting  more  frequent  readi¬ 
ness  evaluations  in  the  face  of  tight  resource  constraints  is  to  use  train¬ 
ing  devices  (e.g.,  simulators,  mockups,  etc.)  instead  of  actual  equipment 
in  the  conduct  of  such  evaluations  (Finley,  Gainer,  &  Muckier,  1974;  Hopkins, 
1975).  In  addition  to  their  intended  uses,  training  devices  can  often  pro¬ 
vide  a  vehicle  for  individual  or  collective  proficiency  evaluations 
(Fitzpatrick  &  Morrison,  1971;  Glaser  &  Klaus,  1972;  Crawford  &  Brock, 

1977).  Historically,  the  most  extensive  uses  of  training  devices  in  pro¬ 
ficiency  assessment  have  been  in  the  aviation  community  (Caro,  1973).  The 
commercial  airlines  and  the  Federal  Aviation  Administration  use  flight  simu¬ 
lators  extensively  in  proficiency  assessment.  Follow-up  studies  have  indi¬ 
cated  that  pilot  performance  in  flight  simulators  is  predictive  of  perform¬ 
ance  in  actual  aircraft  (American  Airlines,  1969;  Weitzman,  Fineberg,  Gade, 

&  Compton,  1979). 


Within  a  military  setting,  the  uses  of  training  devices  in  performance 
evaluation  have  generally  mirrored  civilian  uses  and  primarily  involved  avia¬ 
tion.  There  has  been,  however,  an  increasing  use  of  training  devices  to 
assess  individual  and  collective  proficiency  in  other  areas.  Among  the 
additional  applications  have  been  maintenance  proficiency  an'1  anti-submarine 
warfare  crew  performance.  In  the  Army,  one  long-standing,  non-aviation  pro¬ 
gram  of  individual  and  collective  readiness  assessment  using  a  training  de¬ 
vice  is  found  in  the  Air  Defense  branch.  Here,  the  AN/TPQ-29  simulator 
(and  before  that  the  AN/MPQ-T1  simulator  for  the  Nike-Hercules  system)  is 
used  in  the  conduct  of  operation  readiness  evaluations  for  HAWK  Air  Defense 
units.  The  AN/TPQ-29  is  an  engagement  simulator  (i.e.,  signal  generator) 
capable  of  producing  a  variety  of  simulated  air  defense  combat  situations 
[e,g.,  multiple  targets,  electronic  countermeasures  (ECM)  of  various  kinds, 
etc.].  The  simulator  was  designed  primarily  for  use  as  a  training  device, 
but  it  can  be  (and  is)  used  to  evaluate  individual  and  crew  performance. 

When  using  the  AN/TPQ-29  in  performance  assessment,  an  evaluation  team  loads 
a  "raid  tape"  containing  the  parameters  of  an  air  defense  engagement  into 
the  simulator.  The  HAWK  crew  is  evaluated  on  its  ability  to  defeat  the 
simulated  threat;  performance  checklists  are  used  to  evaluate  individual 
crew  member  performances.  Hardcopy  printouts  of  some  individual  and  crew 
performance  measures  (e.g.,  targets  destroyed,  numbers  of  penetrators,  en¬ 
gagement  tiroes,  operator  reaction  times,  etc.)  are  also  obtainable  from 
the  simulator. 

The  evaluation  of  HAWK  personnel  using  the  AN/TPQ-29  engagement  simu¬ 
lator  illustrates  the  concept  of  a  Training  Device  Operational  Readiness 
Assessment  (Capability,  or  DORAC.  DORAC  simply  means  that  a  proficiency 
assessment  capability  is  included  with  the  training  devices  for  a  materiel 
system.  Once  built  into  the  training  device  system,  the  DORAC  is  used  to 
assess  the  operational  readiness  of  the  individuals  or  crews  that  operate 
the  materiel  system.  As  an  added  feature,  the  measurement  capability  in¬ 
herent  in  the  DORAC  can  also  provide  information  useful  for  other  purposes 
such  as  training  management,  unit  management,  materiel  system  evaluation, 
and  so  forth. 

A  recent  review  of  Army  training  device  proficiency  assessment  capa¬ 
bilities  indicated  that  the  DORAC  principle  can  be  applied  to  the  training 
devices  for  virtually  any  parent  materiel  system  (Shelnutt,  Smillie,  & 

Bercos,  1978).  At  the  present  time,  actual  use  as  in  the  aviation  commun¬ 
ity  or  in  the  HAWK  system  is  somewhat  rare,  but  the  potential  remains. 

There  are,  however,  several  issues  that  remain  to  be  resolved  before  gen¬ 
erally  attempting  to  Implement  the  DORAC  principle  Army-wide.  Among  these 
issues  are:  (1)  the  feasibility  of  the  concept,  and  (2)  its  utility  for 
providing  desired  and  meaningful  data  concerning  individual  or  collective 
performance.  The  objective  of  this  report  is  to  present  the  results  of  an 
analytical  and  empirical  study  carried  out  to  explore  these  issues.  Before 
addressing  the  issues  of  DORAC  utility  and  feasibility,  the  next  portion  of 
the  report  presents  a  brief  overview  of  the  formal  context  for  DORAC  de¬ 
velopment  and  implementation:  the  Army's  training  evaluation  (TE)  system 
and  the  unit  status  reporting  (USR)  system. 


The  Context  for  DORAC  Implementation 


Given  that  the  objective  of  a  DORAC  is  to  provide  decision-makers 
with  readiness  status  and  training  diagnostic  information,  the  formal  con¬ 
text  for  the  collection  and  use  of  the  data  is  the  Army  TE  system  and  the 
USR  system.  Each  of  these  systems  is  described  briefly  as  follows. 

The  Army  Training  and  Evaluation  System.  As  currently  structured 
(SofTech,  Inc,,  1977),  the  Army  TE  system  is  based  upon  the  use  of  Skill 
Qualification  Tests  (SQTs)  and  Army  Training  and  Evaluation  Program  (ARTEP) 
evaluations.  SQTs  are  designed  to  assess  individual  performance  on  criti¬ 
cal  job  tasks.  Tasks  identified  as  critical  and  subject  to  SQT  evaluation 
are  listed  in  Soldier’s  Manuals  (SMs),  which  are  prepared  for  each  Military 
Occupational  Specialty  (MOS) .  Each  job  incumbent  is  provided  with  a  per¬ 
sonal  copy  of  the  SM  for  his/her  MOS. 

The  SQT  test  usually  consists  of  three  portions:  a  performance-oriented 
or  "hands-on"  component  (HOC),  a  written  component,  and  a  job  site  component 
(JSC).  During  the  HOC,  soldiers  are  assessed  on  their  ability  to  perform 
selected  job  tasks  using  actual  equipment.  The  written  portion  of  the  SQT 
is  a  paper-and-pencil  evaluation  of  an  MOS-holder's  job  knowledge.  The  JSC 
covers  designated  skills  which  the  soldier's  supervisor  evaluates  and  certi¬ 
fies  within  the  context  of  the  job  site  (e.g.,  rifle  qualification,  physical 
training,  etc.).  Current  doctrine  calls  for  SQTs  to  be  administered  a  mini¬ 
mum  of  every  two  years. 

Collective  tasks  are  assessed  in  ARTEP  evaluations.  Training  and  eval¬ 
uation  objectives  for  collectives  (e.g.,  platoons,  companies,  battalions) 
are  prescribed  in  ARTEP  publications.  These  publications  are  usually  de¬ 
veloped  by  the  proponent  Training  and  Doctrine  Command  (TRADOC)  schools 
(e.g..  Infantry,  Armor,  Artillery,  Air  Defense,  etc.)  with  the  assistance 
of  the  Army  Training  Support  Center  (ATSC) .  The  actual  ARTEP  evaluation 
is  conducted  within  the  framework  of  a  field  exercise.  Current  doctrine 
calls  for  ARTEPs  to  be  administered  yearly 

It  has  generally  been  recognized  that  a  formal  evaluation  every  year 
or  so  is  not  sufficient  to  address  individual  or  collective  training  prob¬ 
lems  or  indicate  readiness  status.  Accordingly,  units  are  encouraged  to 
conduct  more  frequent  internal  evaluations.  The  results  of  internal  or 
"mini-SQTs"  are  to  be  recorded  in  a  Job  Book  that  lists  individual  profi¬ 
ciency  on  a  task-by-task  basis.  In  addition  to  more  frequent  individual 
evaluations,  units  are  also  encouraged  to  develop  a  Training  and  Evalua¬ 
tion  Outline  (T&EO)  that  prescribes  the  structure  of  an  evaluation  of  their 
collective  proficiency.  Since  the  T&EO  specifies  oolleative  evaluation 
procedures,  it  can  be  classified  as  a  mini-ARTEP. 

In  theory,  unit-level  personnel  use  SQT,  ARTEP,  and  internal  evalua¬ 
tion  results  to  identify  individuals  and  collectives  that  have  identifiable 
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training  deficiencies  (i.e.,  tasks  that  are  not  performed  to  standard). 
Higher  level  commanders  (e.g.,  battalion,  brigade,  division)  review  SQT 
and  ARTEP  results  to  pinpoint  weaknesses  in  their  unit  and  to  identify 
areas  where  additional  training  emphasis  or  assistance  may  be  required. 

At  the  highest  level,  TRADOC  uses  aggregated  SQT  and  ARTEP  results  to 
identify  training  system,  materiel  system,  or  tactical  deficiencies  that 
require  special  emphasis  to  effect  a  solution.  It  is  generally  recognized, 
however,  that  SQTs,  ARTEPs,  and  internal  evaluations  provide  insufficient 
or  too  infrequent  data  to  adequately  serve  these  information  needs. 

The  Unit  Status  Reporting  System.  Army  Regulation  (AR)  220-1  estab¬ 
lishes  the  USR  system  for  "reporting  the  readiness  status  of  selected  ac¬ 
tive  and  reserve  component  units."  USR  reports  are  prepared  on  a  monthly 
basis.  In  the  USRs.  commanders  are  required  to  assess  their  unit  along 
three  specific  dimensions,  and  then  to  combine  the  individual  ratings  into 
an  overall  unit  readiness  rating.  The  individual  dimensions  are: 

1.  Personnel  strength  (operating,  MOS  trained,  and  senior 
grade) . 

2.  Logistics  (actual  vs.  authorized  equipment,  equipment 
status) . 

3.  Training  status. 

The  overall  unit  readiness  rating  is  reported  as  follows: 


Rating 

C— 1 ,  Fully  Ready: 


C-2,  Substantially 
Ready : 


C-3,  Marginally 
Ready : 


Description 

A  unit  fully  capable  of  performing  the 
mission  for  which  it  is  organized  or 
designed.  Unit  may  be  deployed  to  a 
combat  theater  immediately. 

A  unit  has  minor  deficiencies  which 
limit  its  capability  to  accomplish  the 
mission  for  which  it  is  organized  or 
designed.  Unit  may  be  deployed  to  a 
combat  theater  immediately. 

A  unit  has  major  deficiencies  of  such 
magnitude  as  to  limit  severely  its 
capability  to  accomplish  the  mission 
for  which  it  is  organized  or  designed. 
Unit  will  require  a  period  of  intensive 
preparation  before  combat  deployment/ 
employment  except  under  conditions  of 
grave  emergency. 
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Rating  Description 

C-4,  Not  Ready:  A  unit  not  capable  of  performing  the 

mission  for  which  it  is  organized  or 
designed.  Unit  will  require  extensive 
upgrading  prior  to  combat  deployment. 

In  developing  the  USR,  personnel  strength  data  are  taken  from  the 
unit's  SIDPERS  (Standard  Installation/Division  Personnel  System)  report. 
Three  components  influence  the  personnel  rating:  operating  strength,  MOS 
trained  strength,  and  senior  grade  NCO  strength.  Rating  guidelines  for 
each  of  these  components  are  provided  in  the  AR.  Logistics  data  are  ob¬ 
tained  by  comparing  actual  equipment  on  hand  (EOH)  with  Modified  Table  of 
Organization  and  Equipment  authorizations.  Equipment  status  (ES)  is  di¬ 
rectly  obtainable  from  equipment  deadline  reports.  Again,  the  AR  provides 
specific  guidelines  for  assigning  a  unit  logistics  rating  on  the  basis  of 
EOH  and  ES  data. 

Training  status  is  handled  differently  from  either  personnel  strength 
or  logistics.  According  to  AR  220-1,  "The  training  rating  is  a  judtjment 
based  on  an  estimate  of  the  time  required  to  overcutie  training  shortfalls." 
In  making  this  estimate,  the  unit  commander  is  advised  to  consider  SQT 
scores,  ARTEP  results,  and  the  results  of  other  individual  and  collective 
evaluations,  [e.g.,  Emergency  Readiness  Deployment  Exercises  (ERDEs) , 

Field  Training  Exercises  (FTXs) ,  etc.],  but  no  objective  guidelines  for 
their  consideration  are  provided. 

As  a  final  step,  the  commander  selects  an  overall  unit  rating  (C-l 
through  C-4)  that  "best  describes  his  unit's  capability  to  perform  the  mis¬ 
sion  for  which  organized  or  designed."  In  theory,  the  overall  unit  rating 
should  not  exceed  the  lowest  rating  in  a  resource  area  (e.g.,  personnel  or 
logistics)  or  the  training  rating.  For  units  with  major  equipment  systems 
(e.g.,  tanks,  missiles,  artillt  y  pieces,  etc.),  the  percentage  of  total 
equipment  systems  manned  by  trained,  crews  usually  serves  as  a  guide  for 
assigning  the  overall  unit  rating.  For  example,  a  unit  rating  of  C-l 
should  indicate  that  at  least  85%  of  total  systems  are  available;  a  rating 
of  C-2  that  at  least  70%  of  total  systems  are  available;  a  rating  of  C-3 
that  at  least  60%  of  total  systems  are  available;  and  a  rating  of  C-4  that 
the  unit  has  less  than  60%  of  total  systems  available. 


Overview  of  Approach 


The  previous  paragraphs  have  presented  an  overview  of  the  problem 
situation:  obtaining  more  frequent,  reliable,  and  valid  data  concerning 
individual  or  collective  readiness,  while  avoiding  the  expense  of  using 
operational  equipment  to  that  end.  A  proposed  solution  involving  the  use 
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of  training  devices  as  a  proficiei  :y  measurement  bed  (i.e.,  DORAC)  was 
also  presented.  Finally,  the  formal  context  for  operational  readiness 
reporting  and  training  evaluation  was  discussed.  The  remainder  of  the 
report  presents  the  results  of  an  analytical  and  empirical  investigation 
of  the  feasibility  and  utility  of  the  DORAC  concept  as  applied  to  the  prob¬ 
lem  situation. 

In  developing  the  report,  an  orthodox  structure  for  project  feasibil¬ 
ity  analysis  is  employed  (Clifton  &  Fyffe,  1977).  Specifically,  the  study 
considers  three  class  :al  determinants  of  project  feasibility: 

1.  Acceptability.  An  evaluation  of  the  DORAC  concept's  feas¬ 
ibility  in  terms  of  the  end  user,  or  "market".  Obviously, 
in  the  current  context,  a  market  in  the  traditional  usage 
of  the  term  does  uot  exist.  An  analogous  situation  does 
exist,  however,  in  that  there  is  a  constituency  that  must 
accept  the  DORAC  concept  should  it  be  implemented.  Spe¬ 
cific  issues  addressed  under  this  topic  included:  What  is 
the  current  field  situation  in  terms  of  training  evaluation 
and  unit  status  reporting?  Do  field  personnel  perceive  de¬ 
ficiencies  in  current  practices?  If  so,  what  are  they? 

Is  the  DORAC  concept  viewed  as  an  acceptable  solution  to 
any  of  the  reported  deficiencies?  The  vehicle  for  the  con¬ 
duct  of  this  portion  of  the  study  was  a  survey  of  field 
readiness  assessment  (i.e.,  TE  and  USR)  practices.  These 
results  are  presented  and  discussed  in  Section  2. 

2.  Technical  Feasibility.  An  examination  of  the  technical 
aspects  (e.g.,  developmental  considerations,  available  and 
potential  vehicles,  reporting  practices,  etc.)  oT  DORAC 
implementation.  Issues  of  interest  under  this  topic  in¬ 
clude:  Can  DORACs  be  successfully  developed  and  implemented? 
Are  the  results  of  an  implementation  of  the  concept  likely 

to  be  useful7  How  will  DORACs  fit  into  the  training  eval¬ 
uation  and  USR  systems?  Under  what  conditions  is  maximum 
payoff  from  DORAC  implementation  likely  to  occur?  Conclu¬ 
sions  relevant  to  these  issues  were  derived  rationally  on 
the  basis  of  review  of:  (1)  existing  and  projected  Army 
training  devices,  and  (2)  suggested  improvements  to  the  TE 
and  USR  systems  provided  by  survey  participants.  These 
issues  are  addressed  in  Section  3,  Implementation  Issues. 

One  additional  aspect  of  Technical  Feasibility  involves 
the  issue  of  identifying  the  most  cost-effective  capability; 
that  is,  being  able  to  structure  a  DORAC  in  which  the  value 
of  the  readiness  information  provided  offsets  the  costs  en¬ 
tailed  in  obtaining  the  information.  An  analysis  directed 
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at  determining  the  most  cost-effective  DQRAC  is  tenaed  Cost 
and  Information  Effectiveness  Analysis,  or  CIEA  (seu  Hawley 
&  Dawdy,  1981).  To  empirically  investigate  this  is:iue,  a 
demonstration  CIEA  was  performed  on  a  set  of  potential  DORAC 
alternatives  for  the  M16A1  rifle.  The  results  of  this  demon¬ 
stration  CIEA  are  reported  separately  in  Section  4. 

3.  Financial  Feasibility.  An  examination  of  the  resources  re¬ 
quired  to  develop,  implement,  and  sustain  a  system  of  DORACs 
Army-wide.  This  i ssue  is  not  addressed  at  a  detailed  or 
specific  level;  rather  it  is  addressed  at  the  level  of  general 
resource  requirements.  Material  relevant  to  the  issue  of 
financial  feasibility  is  presented  in  Section  3. 

The  final  section  of  the  report  (Section  5)  briefly  summarizes  the 
results  of  the  DORAC  feasibility  and  utility  evaluation.  A  list  of  condi¬ 
tions  under  which  maximum  payoff  from  a  DORAC  is  likely  to  occur  is  also 
presented.  Then,  integrating  these  two  sets  of  results,  specific  recom¬ 
mendations  for  continued  DORAC  developments  are  listed  and  discussed. 
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SECTION  2 


FIELD  READINESS  ASSESSMENT  PRACTICES 
Introduction 


The  first  step  in  establishing  the  feasibility  and  utility  of  the 
DORAC  concept  was  to  determine:  (1)  if  the  intended  end-users  of  DORAC 
information  (i.e.,  field  commanders  and  trainers)  perceive  the  need  for 
improved  performance  assessment,  and  (2)  whether  or  not  the  DORAC  concept 
is  judged  to  be  an  acceptable  solution  to  perceived  evaluation  deficien¬ 
cies.  Since  this  first  portion  of  the  feasibility  analysis  concerns  and- 
user  perceptions,  the  preferred  vehicle  for  obtaining  data  relevant  to 
these  two  issues  was  a  field  survey  of  potentially  affected  units.  The 
survey  addressed  four  sequential  aspects  of  DORAC  feasibility,  listed  as 
follows: 

1.  A  description  of  current  field  reporting  practices.  Con¬ 
sidering  a  cross-section  of  Army  units,  how  are  the  formal 
requirements  of  the  TE  and  USR  systems  interpreted  and 
carried  out? 

2.  The  identification  of  deficiencies  in  the  way  training 
evaluation  and  unit  status  reporting  are  carried  out, 
or  in  the  quality  of  the  information  obtained. 

3.  Suggested  improvements  to  the  training  evaluation  or 
status  reporting  systems.  User  perceptions  regarding 
solutions  to  deficiencies  in  the  TE  or  USR  systems. 

4.  DORAC  concept  acceptability  and  perceived  utility.  End- 
user  opinions  concerning  the  acceptability  of  the  DORAC 
concept,  and  their  comments  concerning  the  utility  of 
obtaining  more  frequent  or  higher  quality  performance  data. 

As  noted  previously,  a  major  component  of  DORAC  feasibility  is  the 
reaction  of  the  intended  users  of  the  information.  Realistically  speaking, 
any  performance  evaluation  program,  no  matter  how  well  intended,  has  the 
potential  for  being  viewed  as  an  assessment  of  unit  commanders.  Field- 
level  rejection  of  the  concept  will  lead  to  DORAC  being  viewed  as  just 
another,  and  perhaps  more  expensive,  evaluation  system  that  functions  no 
better  than  the  method  it  replaces  or  supplements.  In  order  to  reduce  the 
likelihood  of  this  outcome,  it  was  judged  necessary  to  gain  some  insight 
into  the  probable  reception  a  DORAC  would  receive.  End-user  opinions  re¬ 
garding  potential  uses  of  data  and  the  mode  of  application  for  DORAC  can 
have  a  bearing  on  the  eventual  acceptance  of  the  concept. 


2-1 


Approach 


The  vehicle  for  obtaining  information  concerning  the  four  aspects  of 
DORAC  feasibility  cited  above  was  a  field  survey.  The  survey  was  conducted 
with  personnel  from  a  cross-section  of  units  representative  of  those  likely 
to  be  affected  by  a  decision  to  implement  DORAC  Army-wide.  Units  selected 
for  participation  in  the  field  survey  were  located  at  Ft.  Benning,  GA; 

Ft.  Knox,  KY;  and  Ft.  Bliss,  TX.  At  these  posts,  survey  participants  were 
selected  from  the  following  units: 

Unit  Location 


1. 

4/37  Armor 

Ft. 

Knox 

2. 

4/51  Mech.  Infantry 

Ft. 

Knox 

3. 

3/3  Field  Artillery 

Ft. 

Knox 

4. 

2/10  Field  Artillery 

Ft. 

Benning 

5. 

1/58  Infantry 

Ft. 

Benning 

6. 

2/55  Air  Defense  Artillery 
(ADA) 

Ft. 

Bliss 

7. 

1/7  ADA 

Ft. 

Bliss 

8. 

1/3  Armored  Cavalry 

Regiment  (ACR) 

Ft. 

Bliss 

9. 

3/3  ACR 

Ft. 

Bliss 

Survey  information  was  obtained  in  face-to-face  interviews  with  personnel 
representing  a  cross-section  of  unit  perspectives  (e.g.,  command  vs.  staff, 
brigade  vs.  battalion  vs.  company). 

The  first  step  in  describing  field  proficiency  assessment  and  status 
reporting  practices  was  to  develop  suitable  survey  instrumentation.  Survey 
instrument  construction  was  conducted  in  two  steps: 

1.  Development  and  tryout  of  a  preliminary  interview  guide. 

2.  Development  of  refined  survey  instruments  appropriate 
for  various  categories  of  interviewees. 

The  initial  interview  guide  (see  Appendix  A)  consisted  of  a  set  of 
preliminary,  general  questions  designed  to  establish  the  range  and  type 
of  potential  responses.  This  initial  interview  guide  was  used  with  per¬ 
sonnel  from  2/55  ADA  at  Ft.  Bliss,  TX. 
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Based  on  the  preliminary  survey  results,  the  initial  interview  guide 
was  refined  into  two  separate  and  more  detailed  interview  guides,  one  for 
staff  level  (e.g.,  brigade,  battalion)  personnel,  and  another  for  company/ 
battery/troop  level  personnel.  The  resulting  detailed  interview  guides 
are  presented  as  Appendixes  B  and  C.  Following  the  development  of  the 
detailed  interview  guides,  unit  surveys  were  conducted  in  the  time  inter¬ 
val  20  February  to  3  April  1980. 


Relevant  Field  Survey  Results 


The  most  striking  result  of  the  unit  survey  was  the  wide  range  of 
evaluation  activities  conducted  by  field  personnel.  In  addition  to  re¬ 
quired  SQT  and  ARTEP  evaluations,  units  in  the  various  branches  conduct 
numerous  formal  and  informal  evaluations  tailored  to  the  specific  demands 
of  the  materiel  systems  they  employ.  Many  respondents  indicated  that  a 
large  amount  of  their  training  time  is  spent  "peaking"  for  these  evalua¬ 
tions.  In  this  context,  peaking  refers  to  the  practice  of  intensively 
tuning-up,  or  cramming,  for  an  upcoming  evaluation. 

Both  the  SQT  and  the  ARTEP  generally  are  conducted  much  as  formally 
prescribed.  As  one  respondent  noted,  "the  unit  does  what  it  has  to  do." 

The  SQT  drew  mixed  reviews  from  survey  participants.  Some  respondents 
indicated  that  the  SQT  does  a  good  job  of  evaluating  necessary  job  skills; 
other  respondents  were  less  positive.  A  fairly  widespread  view  was  that 
the  written  portion  of  the  SQT  places  too  much  emphasis  on  reading  ability. 
Personnel  who  are  judged  to  be  adequate  job  performers  often  do  not  per¬ 
form  well  on  the  written  portion  of  SQT  because  they  do  not  read  well. 
Respondents  were  unanimous  on  the  importance  of  having  competent  eval¬ 
uators  for  the  hands-on  portions  of  the  SQT.  Also,  survey  participants 
universally  favored  the  idea  of  hands-on,  performance-oriented  proficiency 
evaluation. 

In  terms  of  the  internal  or  raini-SQT,  most  of  the  units  visited  con¬ 
duct  some  type  of  informal  proficiency  evaluation.  These  evaluations  are 
usually  carried  out  by  crew  chiefs  or  squad  leaders  on  what  was  referred 
to  as  a  task  sampling  basis;  that  is,  on  a  daily  or  weekly  basis,  a  sample 
of  tasks  for  evaluation  are  selected  from  the  SM.  Most  respondents  ad¬ 
mitted  that  these  internal  evaluations  are  of  limited  utility  because 
there  are  no  standardized  procedures  for  their  conduct  and  performance 
standards  are  often  not  vigorously  enforced.  The  Job  Book  scheme  for  re¬ 
cording  the  results  of  internal  SQTs  is  apparently  not  working.  For  many 
MOSs,  Job  Books  are  not  available.  Where  Job  Books  are  available,  they 
are  often  not  used.  None  of  the  units  contacted  placed  a  high  priority 
on  keeping  Job  Books  up-to-date.  Several  respondents  attributed  this 
situation  to  a  lack  of  command  interest.  Job  Books  are  considered  to  be 
"just  another  paper  requirement"  with  little  perceived  utility. 
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Respondent  comments  concerning  the  ARTEP  were  also  mixed.  For  exam¬ 
ple,  the  Armored  personnel  contacted  thought  their  ARTEP  was  very  good; 
the  Air  Defense  units  surveyed  no  longer  conduct  an  ARTEP  because  theirs 
had  proven  unsatisfactory  in  practice.  The  Infantry  units  queried  rated 
the  quality  of  their  ARTEP  between  these  two  extremes.  The  internal  or 
mini-ARTEFs  that  had  been  developed  and  tried  were  characterized  as  loosely 
constructed  and  non-standardized.  A  primary  criticism  of  both  types  of 
ARTEP  evaluation  concerned  a  general  lack  of  combat-referenced  performance 
standards.  Respondents  generally  agreed  that  a  well-conducted  ARTEP  would 
be  the  preferred  method  of  assessing  overall  unit  combat  readiness.  The 
ARTEP  permits  all  portions  of  the  unit  to  carry  out  all  aspects  of  their 
mission.  In  addition,  a  well  developed  ARTEP  would  exercise  the  command, 
control,  and  communication  elements  that  are  missing  from  more  piecemeal 
evaluation  procedures. 

The  most  important  aspect  of  characterizing  current  proficiency  assess¬ 
ment  practices  concerned  how  units  determine  their  training  readiness  level 
for  the  USR.  When  asked  to  address  this  subject,  survey  respondents  gen¬ 
erally  gave  one  quick  reply:  "very  subjectively."  The  training  status 
rating  was  judged  to  be  the  most  subjective  of  the  three  component  ratings 
(personnel,  equipment,  training).  As  one  respondent  phrased  it:  "we  have 
no  real  way  to  tell  what  our  proficiency  level  is."  Another  respondent 
stated  that  the  overall  rating  is  "primarily  driven  by  equipment  and  per¬ 
sonnel  strength  considerations";  this  respondent  also  admitted  that  "the 
training  rating  never  drives  it  (the  overall  rating)  down." 

In  summary,  the  first  and  second  issues,  current  field  proficiency 
assessment  practices  and  perceived  deficiencies,  are  characterized  by  the 
following  points: 

1.  Units  carry  out  whatever  evaluation  requirements  are  placed 
on  them  from  higher  headquarters. 

2.  Formal,  explicit  evaluation  requirements  are  met;  suggested 
or  recommended  evaluation  activities  are  usually  given  a 
lower  priority. 

3.  Both  the  TE  and  the  USR  systems  are  plagued  by  a  lack  of 
standardization  and  performance  standards. 

4.  Unit  personnel  believe  that  they  are  in  the  best  position 
to  judge  their  unit's  proficiency.  They  admit,  however, 
that  internally  developed  or  conducted  evaluations  are 
usually  quite  deficient. 

5.  The  USR  is  acknowledged  to  be  "subjectively  distorted." 

The  training  rating  portion  of  the  USR  is  the  most  sub¬ 
jective  component  of  the  three  factors  considered. 
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In  terms  of  suggestions  regarding  improvements  to  the  current  TE  and 
USR  systems,  the  survey  results  were  again  mixed.  Some  respondents  judged 
the  present  systems  to  be  adequate.  Other  respondents  were  of  the  opin¬ 
ion  that  the  current  system  places  too  much  emphasis  on  numbers,  or 
quantitative  results.  These  respondents  stated  that  an  emphasis  on 
quantitative  results  produces  a  "we-they"  situation  in  which  a  unit's 
emphasis  is  on  "passing  the  test"  rather  than  on  identifying  and  correct¬ 
ing  performance  deficiencies.  Specific  recommendations  for  improving  the 
TE  and  USR  systems  included  the  following: 

Conduct  integrated  unit  evaluations  (e.g.,  ARTEP,  FTX,  etc.). 

The  current  assessment  system  is  too  piecemeal. 

Provide  more  detail  on  evaluation  requirements,  down  to  the 

task  level. 

Conduct  more  field  exercises. 

Provide  for  more  time  on  equipment. 

Provide  more  immediate  feedback  on  SQT. 

Make  training  and  evaluation  activities  more  realistic. 

*  Provide  for  evaluation  by  outsiders. 

An  additional  implementation  issue  was  the  acceptability  and  per¬ 
ceived  utility  of  the  DORAC  concept.  This  issue  was  assessed  by  two 
questions  on  the  interview  guide: 

Q.  What  is  your  opinion  of  using  trainers  or  simulators 
vn  training  evaluation? 

Q.  For  the  training  portion  of  your  USHs  could  you  use  skills 
evaluation  data  which  have  been  generated  by  trainers/ 
simulators?  How  could  you  use  such  information ,  and 
what  is  your  attitude  concerning  its  use? 

In  reviewing  the  field  survey  results,  some  striking  response  disparities 
to  these  questions  were  apparent.  The  primary  factor  determining  positive 
or  negative  responses  appeared  to  be  the  respondents'  previous  experience 
with  training  devices.  Air  Defense  personnel,  through  their  experience 
with  the  AN/MPQ-T1  and  AN/TPQ-29,  were  generally  receptive  to  the  idea  of 
using  training  devices  for  evaluation  purposes.  One  respondent  stated: 

"In  Air  Defense,  we  could  not  do  without  them."  Not  surprisingly.  Air 
Defense  personnel  also  were  not  opposed  to  the  idea  of  usi*^  training  de¬ 
vice  derived  performance  data  as  a  guide  for  the  USR  training  status  rating. 
In  fact,  one  HAWK  battalion 'commander  stated  that  AN/TPQ-29  results  are 
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considered  in  assigning  the  training  rating.  The  only  recurring  complaint 
expressed  by  Air  Defense  personnel  concerned  equipment  density ;  there 
simply  are  not  enough  devices  available  to  accommodate  all  units. 

Survey  results  from  other  units,  particularly  Armor,  presented  an  en¬ 
tirely  different  viewpoint.  Many  of  the  Armor  respondents  characterized 
their  training  devices  as  "substandard"  and  "inferior  to  actual  equipment". 
One  respondent  stated  flatly:  "the  devices  don't  work."  As  a  result  of 
these  negative  views  toward  training  devices  in  general,  most  of  the  non- 
Air  Defense  respondents  also  were  opposed  to  the  idea  of  using  training 
device  generated  performance  data  in  the  training  portion  of  the  USR.  Some 
typical  comments  on  the.  issue  included  the  following: 

Based  on  my  experience  with  current  devices,  no. 

The  current  devices  cannot  be  used. 

Not  with  present  simulators.  With  adequate  simulators,  maybe. 

People  don't  get  serious  with  simulators. 

Partial  inclusion  might  be  O.K. 

No,  period. 

Perceptions  of  current  devices  would  not  allow  the  results 

to  drive  USR  training  ratings  (comment  of  battalion  commander). 

Not  surprisingly,  the  above  results  indicate  that  personnel  who  have 
had  favorable  experiences  with  training  devices  (e.g.,  the  AN/TPQ-29)  view 
them  favorably  and  do  not  have  a  negative  attitude  toward  using  the  de¬ 
vices  for  proficiency  evaluation.  On  the  other  hand,  the  experiences  of 
many  of  the  respondents  with  training  devices  have  been  negative.  Train¬ 
ing  devices  were  characterized  as  being  inferior,  sub-standard,  and  of 
limited  utility.  Accordingly,  these  personnel  are  also  opposed  to  the 
use  of  training  devices  in  performance  evaluation. 


Discussion 


The  results  presented  in  the  previous  paragraphs  indicate  that  field 
personnel  do  recognize  deficiencies  in  the  current  TE  and  USR  systems. 
Among  the  major  reported  deficiencies  are  the  following: 

1.  A  general  lack  of  standardization  in  evaluation  procedures 
for  all  but  SQTs. 

2.  A  general  lack  of  combat-referenced  performance  standards. 
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3.  For  many  materiel  systems,  there  is  no  sure  means  for  ob¬ 
taining  acceptable  performance  data. 

4.  There  is  a  generally  recognized  lack  of  objectivity  in 
the  way  USR  ratings  are  assigned. 

When  queried  concerning  potential  solutions  to  the  deficiencies  cited 
above,  survey  respondents  listed  the  following  suggestions: 

1.  Provide  for  realism  in  training  and  evaluation. 

2.  Conduct  integrated  unit  evaluations. 

3.  Provide  immediate  performance  feedback. 

4.  Provide  for  evaluation  by  (impartial)  outsiders. 

Obviously,  some  of  these  suggestions  are  solutions  to  other  problems, 
rather  than  to  the  identified  problems,  and  require  a  priori  that: 

(1)  evaluation  procedures  are  standardized,  (2)  adequate  performance 
standards  exist,  and  (3)  a  vehicle  for  obtaining  proficiency  data  is 
available. 

The  survey  results  also  indicated  that  training  devices  are  gener¬ 
ally  not  pevaeivcd  to  fit  into  any  of  the  proposed  solutions.  This  atti¬ 
tude  primarily  resulted  from  previous  bad  experiences  with  training  de¬ 
vices  on  the  part  of  many  respondents.  Given  the  favorable  response  of 
the  Air  Defense  personnel,  an  argument  could  be  made  to  the  effect  that 
the  DORAC  concept  can  be  made  acceptable  to  field  personnel,  but  much  con¬ 
vincing  would  be  required.  Considerable  "bad  press"  associated  with  train¬ 
ing  devices  will  have  to  be  overcome.  The  results  suggest  that  if  th, 

DORAC  concept  is  adopted  it  should  be  implemented  deliberately  and  where 
the  likelihood  ot  visible  success  is  high.  Then,  after  a  successful  eval¬ 
uation  program  has  been  developed,  the  concept  should  be  publicized 
throughout  the  Army. 

A  second  issue  relevant  to  the  topic  of  DORAC  acceptability  concerns 
a  general  aversion  to  quantitative  evaluation  on  the  part  of  field  per¬ 
sonnel.  During  the  field  survey,  some  respondeiits  indies  '.ed  that  they 
would  readily  accept  and,  in  fact,  welcome  any  information  that  might  be 
provided  by  a  DORAC  if  the  data  were  not  reported  as  a  formal  part  of  the 
USR.  As  noted  previously,  any  performance  assessment  program  has  the  po¬ 
tential  for  being  viewed  as  an  evaluation  of  unit  commanders .  The  natural, 
tendency  of  commanders  in  such  cases  will  be  to  protect  their  careers  by 
making  the  evaluation  as  non- threatening  as  possible  (one  of  the  survey 
respondents  referred  to  this  outcome  as  "going  the  way  of  the  OER" — 

Officer  Efficiency  Report).  It  is  doubtful  that  any  objective  performance 
evaluation  program  can  be  implemented  successfully  if  field  commanders  view 
it  as  a  potential  threat.  How  this  aversion  to  quantitative  evaluation 
can  be  rationalized  with  the  intent  of  DORAC  remains  an  empirical  issue. 
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In  the  final  analysis,  it  is  obvious  that  most  field  personnel  will 
not  greet  the  DORAC  concept  with  enthusiasm.  Previous  bad  experiences 
with  training  devices  and  an  aversion  to  quantitative  evaluation  will 
likely  combine  to  produce  resistance,  perhaps  passive,  to  DORAC  imple- 
mentatioii.  Assuming,  however,  that  implementation  is  conducted  properly 
and  carefully,  the  question  of  whether  or  not  DORAC  will  be  accepted  by 
field  personnel  can  be  answered  with  a  qualified  "yes".  The  current  atti¬ 
tude  of  Air  Defense  personnel  strongly  supports  this  position.  It  should 
be  noted,  however,  that  this  positive  attitude  is  the  result  of  more  than 
20  years  of  positive  experiences  with  training  devices  in  evaluation. 


2-8 


SECTION  3 


IMPLEMENTATION  ISSUES 
Introduction 


Given  an  affirmative  answer  to  the  issue  of  DORAC  concept  accepta¬ 
bility,  the  next  two  aspects  of  feasibility  are  Technical  and  Financial 
feasibility.  Technical  feasibility  involves  determining  whether  or  not 
it  is  possible  to  provide  required  measurement  capabilities  using  current 
or  projected  training  devices.  The  output  of  the  technical  feasibility 
analysis  then  provides  the  structure  necessary  to  outline  the  resource 
committment  required  to  produce  the  technical  capability.  In  this  manner, 
the  technical  analysis  provides  the  framework  for  the  Financial  feasibil¬ 
ity  analysis. 

Under  the  heading  of  technical  feasibility,  three  questions  are  at 
issue : 

1.  Do  vehicles  (i.e.,  training  devices)  currently  exist  to 
support  the  DORAC  concept,  and  what  are  the  prospects  for 
the  future? 

2.  What  will  be  required  to  develop  DORACs  in  present  and 
future  training  devices? 

3.  VJhat  will  be  required  to  implement  the  DORAC  concept 
Army-wide? 

Once  the  above  issues  have  been  addressed,  the  resources  required  to  de¬ 
velop  the  DORAC  concept  can  be  projected.  At  this  stage  of  the  concept 
study,  resource  requirements  were  considered  at  a  very  general  level; 
that  is,  categories  and  types  of  resource  requirements  are  identified 
and  discussed.  Later,  in  Section  4,  the  resource  requirements  associated 
with  a  specific  DORAC  are  considered. 


Technical  Feasibility 


The  first  issue  under  the  topic  of  technical  feasibility  concerns 
whether  or  not  vehicles  for  DORAC  implementation  currently  exist.  In 
addition,  the  prospects  for  the  future  are  also  reviewed.  This  issue 
can  be  addressed  quite  directly  by  referring  to  DA  Pamphlet  310-12,  Index 
and  Description  of  Army  Training  Devices,  and  the  ATSC  publication  Compre¬ 
hensive  Plan  for  Training  Devices. 
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DA.  Pamphlet  310-12  lists  all  of  the  training  devices  currently  in 
the  Army  inventory.  The  list  runs  literally  into  hundreds  of  devices. 

These  devices  range  in  complexity  from  simple  artillery  noise  simulators 
to  quite  sophisticated  items  such  as  the  Guided  Missile  System  Radar  Sig¬ 
nal  Simulator  Station  (AN/TPQ-29)  and  several  aviation  cockpit  simulators. 
In  addition  to  the  fielded  devices,  the  comprehensive  plan  for  training 
devices  presents  a  long  list  of  training  devices  currently  undergoing 
development.  Again,  the  projected  devices  present  a  considerable  range 
in  complexity. 

It  would  thus  seem  that  there  is  no  shortage  of  candidates  for  DORAC 
development.  Available  candidates,  however,  do  not  imply  that  DORACs  can 
be  developed  and  implemented.  As  Shelnutt  e£  al  (1978)  note,  the  cost 
of  retrofitting  a  measurement  capability  to  an  existing  training  device 
is  likely  to  be  quite  high.  Thus,  for  current  training  devices  that  do 
not  already  have  some  inherent  performance  assessment  capability  (i.e., 
the  AN/TPQ-29)  the  only  cost-effective  course  of  action  may  be  to  employ 
a  human-based  evaluation  system  (i.e.,  checklist,  rating  scale,  etc.), 
even  though  such  a  measurement  system  may  not  provide  data  of  the  high¬ 
est  quality.  In  the  present  context,  these  person-based,  add-on  systems 
are  denoted  as  "quasi-DORACs".  Quasi-DORACs  can  be  developed  for  nearly 
any  training  device  that  permits  a  simulation  of  all  or  part  of  an  indi¬ 
vidual/collective  mission. 

The  most  obvious  candidates  for  the  development  of  more  sophisticated 
(i.e.,  "hard")  DORACs  are  training  devices  that  include  some  kind  of  in¬ 
formation  processing  capability,  or  weapons  systems  that  include  an  in¬ 
formation  processing  capability  that  can  accommodate  a  proficiency  mea¬ 
surement  add-on  (e.g.,  many  missile  systems).  A  review  of  the  inventory 
of  current  training  devices  reveals  several  such  systems/devices.  Most 
of  them,  however,  do  not  have  extensive  proficiency  assessment  capabilities 
currently  built  in.  Thus,  some  amount  of  retrofitting  would  be  required. 
The  list  of  future  training  devices  reveals  potentially  more  candidates 
for  hard  DORACs.  This  higher  number  reflects  an  increasing  use  of  materiel 
systems  having  an  inherent  data  processing  capability.  From  the  descrip¬ 
tions  provided,  it  appears  that  many  of  these  projected  devices  are  well 
along  in  their  developmental  cycle  and  may  also  require  extensive  retro¬ 
fitting  or  design  modification  in  order  to  accommodate  more  than  a  quasi 
proficiency  assessment  capability. 

The  potential  for  hard  DORAC  development  in  developing  training  de¬ 
vices  may  not  be  as  bleak  as  the  previous  paragraph  seems  to  indicate. 

For  example,  th-».  U.S.  Army  Research  Institute  Field  Unit  at  Ft.  Bliss  is 
currently  sponsoring  a  project  titled  Optimizing  Operator  Performance  on 
Advanced  Training  Simulators.  A  portion  of  this  effort  involves  develop¬ 
ing  an  operator  performance  recording,  play-back,  and  scoring  capability 
for  the  PATRIOT  Tactical  Operations  Simulator/Trainer  (TOS/T).  Although 
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the  TOS/T  was  not  designed  to  accommodate  a  proficiency  assessment  capa¬ 
bility,  modifying  the  simulation  software  to  record  and  score  operator  per¬ 
formance  did  not  present  a  particularly  difficult  problem.  The  simulation 
software  was  modified  to  accommodate  what  could  be  termed  a  hard  DORAC 
capability  without  modifying  any  of  the  system  hardware  and  without  inter¬ 
fering  with  on-going  TOS/T  research  or  evaluation  activities. 

In  view  of  the  above  results,  it  appears  safe  to  state  that  a  large 
potential  for  DORAC  implementation  currently  exists.  Furthermore,  the  fu¬ 
ture  looks  even  more  promising  in  terms  of  the  number  of  training  devices 
having  DORAC  potential. 

Although  many  training  devices  having  DORAC  potential  currently  exist 
and  others  will  come  on-line  in  the  future,  potential  is  not  sufficient  to 
ensure  the  successful  application  of  the  DORAC  concept  throughout  the  Army. 
In  order  to  be  useful  and  successful,  the  training  devices  with  DORAC  will 
have  to  be  made  available  in  sufficient  quantities  to  meet  field  training 
and  evaluation  requirements.  Also,  drawing  upon  the  results  of  the  field 
survey,  the  training  devices  will  have  to  be  of  high  quality  and  will  have 
to  be  supported.  In  this  context,  support  refers  to  providing  a  suffi¬ 
cient  number  of  operators  and  evaluators,  and  to  insuring  the  availability 
of  adequate  maintenance  and  other  peripheral  support  facilities. 

In  addition  to  the  obvious  support  requirements  (i.e.,  adequate  num¬ 
bers  of  devices,  maintenance  support,  etc.),  some  additional  background 
work  will  also  be  required.  This  will  involve  developing  standard  oper¬ 
ating  procedures  (SOPs)  for  the  conduct  of  DORAC  evaluation  and  providing 
realistic,  combat-referenced  performance  standards  for  DORAC  evaluations. 
Judging  from  the  field  survey  results,  the  dual  issues  of  standardization 
and  realistic  performance  standards  are  critical  to  the  success  of  the 
DORAC  concept. 

Once  the  capability  has  been  developed  and  adequately  supported,  the 
final  technical  issue  concerns  how  the  performance  data  should  be  used. 

It  appears  that  the  formal  TE  system  is  adequate  to  accommodate  DORAC  in¬ 
formation.  DORACs  can  provide  a  full  or  partial  vehicle  for  the  conduct 
of  a  wide  variety  of  internal  and  external  evaluation  activities.  The 
Unit  Status  Reporting  (USR)  system  would,  however,  have  to  be  modified 
to  provide  for  the  inclusion  of  standardized  training  status  data.  The 
use  of  standardization  teams  to  promote  objectivity  in  the  USR  process 
would  also  be  desirable.  Operational  readiness  evaluation  procedures 
currently  employed  by  the  U.S.  Air. .Force  and  in  the  Army  aviation  commun¬ 
ity  (for  example,  see  TC-1-134  or  TC-1-135)  provide  some  insight  into  the 
potential  structure  of  such  a  modified  USR  system. 

In  summary,  the  DORAC  concept  is  technically  feasible.  However, 
several  actions  on  the  part  of  the  Army  must  be  undertaken  before  attempt¬ 
ing  to  implement  the  concept  Army-wide.  These  actions  are  listed  as 
follows: 
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1.  The  Army’s  deployment  practices  for  training  devices  will 
have  to  be  upgraded.  This  will  involve  increasinj  device 
density,  increasing  device  availability,  and  insuring  the 
availability  of  personnel  trained  to  operate  and  repair 
the  devices.  In  addition,  it  is  essential  that  training 
device  deployment  not  seriously  lag  that  of  the  materiel 
system. 

2.  Standard  operating  procedures  for  DORAC  evaluations  must 
be  developed  and  enforced. 

3.  Realistic,  combat-referenced  performance  standards  for  DORAC 
evaluations  must  be  provided.  Units  also  must  be  provided 
with  guidelines  on  how  to  use  DORAC  generated  proficiency 
status  information  to  increase  training  efficiency. 

4.  The  USR  system  should  be  modified  to  accommodate  standardized 
individual/unit  proficiency  information. 


Financial  Feasibility 

The  final  aspect  of  DORAC  concept  feasibility  concerns  the  resource 
commitment  on  the  part  of  the  Army  that  will  be  necessary  to  develop,  im¬ 
plement,  and  support  a  capability.  Many  of  the  factors  that  will  impact 
upon  the  eventual  cost  of  a  system  of  DORACs  have  already  been  noted.  In 
the  interest  of  completeness,  however,  these  factors  are  listed  again  and 
elaborated  upon.  The  individual  resource  factors  considered  are  listed 
as  follows: 

1.  Facilities  and  Equipment 

2.  Materials 

3.  Personnel 

Overall,  the  additional  costs  of  developing  and  implementing  a  DORAC, 
where  the  evaluation  mission  is  additional  to  the  basic  training  mission, 
are  anticipated  to  be  minimal.  Of  these  costs,  facilities  and  equipment 
will  likely  be  the  single  most  costly  aspect  of  a  decision  to  implement 
DORAC  Army-wide.  First  of  all,  a  dens  if"  equipment  sufficient  to 

accommodate  all  planned  evaluations  which  ..  ed  to  be  accomplished  from  the 
conduct  of  training,  if  any,  must  be  provided.  Judging  from  the  survey 
results,  this  will,  i'  some  cases,  require  a  higher  density  of  training 
devices  than  has  currently  been  provided.  Along  with  an  adequate  device 
density,  appropriate  support  equipment  and  facilities  (i.e.,  land,  build¬ 
ings,  etc.)  will  also  be  required.  Higher  equipment  densities  will  mean 
more  support  facilities  than  are  now  provided. 
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A  second  aspect  of  DORAC  support  is  training  and  evaluation  materials. 
Material  support  will  run  the  gamut  from  computer  software,  to  films,  to 
printed  matter,  to  miscellaneous  items  such  as  paper  and  pencils.  The  po¬ 
tential  cost  of  this  aspect  of  DORAC  support  should  not  be  underestimated. 
For  example,  computer  software  can  cost  thousands  of  dollars  to  develop; 
scenario  films  for  use  in  the  Air  Defense's  Moving  Target  Simulator  (MTS) 
cost  in  the  neighborhood  of  $3-to-4G0,000  to  develop;  scenarios  for  use 
with  the  PATRIOT  TOS/T  are  very  costly  and  time  consuming  to  produce.  Not 
all  DORACs  will  impose  resource  demands  at  the  levels  cited  above,  but  the 
potential  for  costly  materials  support  exists. 

The  third  aspect  of  financial  feasibility  is  personnel.  In  this  area, 
DORAC  resource  demands  are  likely  to  vary  considerably.  Some  DORACs  will 
require  little  more  than  a  part-time  evaluator/maintainer  drawn  from  an 
affected  unit.  Other  systems  may  require  full-time  evaluators,  operators, 
and  maintenance  personnel.  For  example,  the  PATRIOT  TOS/T,  admittedly  a 
very  complex  training  device,  requires  a  full-time  complement  of  3-to-A 
operators  and  software  maintenance  personnel  (both  military  and  contractor) 
to  keep  it  operational.  In  addition,  hardware  maintenance  support  is  pro¬ 
vided  on  a  contractual  basis. 

No  matter  what  level  or  type  of  support  is  required,  one  fact  is 
apparent  from  the  field  survey  results:  if  the  DORAC  concept  is  imple¬ 
mented,  it  must  be  given  adequate  support.  Without  adequate  provisions 
for  support,  DORAC  will  be  doomed  to  failure  in  terms  of  meeting  the  ob¬ 
jectives  for  which  it  is  intended. 
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SECTION  4 


DEMONSTRATION  OF  ANALYSIS  METHOD  FEASIBILITY  AND  DORAC  UTILITY 

Introduction 


Section  4  is  concerned  with  the  conduct  of  a  demonstration  Cost  and 
Information  Effectiveness  Analysis  (CIEA)  performed  on  a  set  of  hypo¬ 
thetical  DORAC  alternatives.  CIEA  is  a  cost-benefit  trade-off  analysis 
similar  in  nature  to  cost  and  training  effectiveness  analysis  (CTEA)  and 
to  cost  and  operational  effectiveness  analysis  (COEA).  The  objective  of 
CIEA  is  to  provide  the  framework  for  selecting  a  preferred  DORAC  alterna¬ 
tive  in  terms  of  the  value  of  the  information  provided  (e.g.,  operational 
readiness,  training  status,  etc.)  versus  the  cost  of  developing,  imple¬ 
menting,  and  operating  the  capability.  CIEA  is  intended  to  serve  as  a 
guide  to  decision-makers  in  developing  and  evaluating  alternative  DORAC 
concepts  for  any  parent  materiel  system. 

As  a  methodology,  CIEA  (along  with  CTEA  and  COEA)  is  a  member  of  a 
set  of  procedures  generally  known  as  cost-effectiveness  analysis.  The 
term  cost-effectiveness  analysis  denotes  a  procedure  in  which  alternative 
system  concepts  are  evaluated  using  measures  of  cost  (usually  dollars) 
and  separate  measures  of  effectiveness  (e.g.,  reliability,  speed,  prob¬ 
ability  of  accomplishing  a  task,  or  a  weighted  index  of  a  number  of  such 
factors)  (Bar Ish  &  Kaplan,  1978).  Under  this  approach,  cost  and  effec¬ 
tiveness  values  for  each  alternative  are  determined.  The  systems  are 
then  evaluated  on  the  basis  of  whether  the  added  benefits  of  the  more 
effective  alternatives  are  worth  their  added  costs.  The  use  of  cost- 
effectiveness  analysis  is  common  in  the  evaluation  of  military  materiel 
and  training  systems  (for  example,  see  TRADOC  Pam  11-8  or  TRADOC  Pam  71-10). 

The  Statement  of  Work  (SOW)  for  the  current  project  calls  for  a  pre¬ 
liminary  CIEA  methodology  to  be  developed  and  then  demonstrated  using  a 
training  device  system  selected  by  the  Army  Research  Institute  (ARI)  in 
conjunction  with  the  Project  Manager  for  Training  Devices  (PM  TRADE) 
and  the  Army  Training  Support  Center  (ATSC).  These  devices  were  to  serve 
as  a  research  vehicle  for  CIEA  development,  a  demonstration  of  method 
usability,  and  to  provide  an  evaluation  and  possible  demonstration  of 
DORAC  utility  in  the  selected  cases.  The  obj  tive  of  Section  4  is  to 
serve  the  last  two  purposes  by  presenting  the  'ults  of  an  application 
of  the  preliminary  CIEA  methodology  (denoted  MAUn-CIEA,  MAUM  standing  for 
multi'’ ttribute  utility  measurement)  (see  Hawley  &  Dawdy,  1981)  to  the 
selected  training  device  and  a  baseline  readiness  evaluation  procedure. 

The  training  devices  selected  for  actual  use  in  the  demonstration 
CIEA  are  applicable  to  the  Army's  standard-issue  rifle,  the  M16A1.  These 
training  devices  are  designated  the  Weaponeer  and  the  Squad  Weapons 
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Analytical  Trainer;  or  SWAT.  Each  of  tne  subject  training  devices  is  de¬ 
scribed  in  additional  detail  later  in  this  section  of  the  report.  For 
evaluation  purposes,  the  training  devices  are  structured,  alone  and  in 
combination,  to  constitute  a  set  of  feasible  DORAC  alternatives.  The  al¬ 
ternatives  are  to  be  evaluated  with  respect  to  the  utility  of  the  informa¬ 
tion  they  provide  versus  the  cost  required  to  obtain  it.  Alternatives 
are  evaluated  against  each  other  and  against  a  baseline  condition  defined 
as  Record  Fire  (RF)  conducted  one  time  per  year.  RF  is  the  vehicle 
presently  used  to  assess  individual  marksmanship  proficiency  on  the  M16A1 
rifle . 

The  following  portions  of  th  .s  section  carry  a  set  of  proposed  M16A1 
DORACs  through  the  steps  of  a  CIEA  evaluation.  As  outlined  in  Hawley  and 
Dawdy  (1981),  the  phases,  and  nested  steps,  of  the  MAUM-C1EA  methodology 
are  listed  as  follows: 

1.  Assess  needs  and  constraints. 

2.  Define  system  objectives 

a.  Determine  issues  tc  which  utility  ratings  are  relevant. 

b.  Determine  relevant  factors  on  which  alternative 
system  concepts  are  to  be  assessed. 

c.  Identify  perspective  from  which  utility  is  to  be 
assessed. 

3.  Identify  operational  requirements  for  the  attainment  of 
system  objectives. 

A.  Develop  alternative  system  concepts. 

5.  Establish  system  evaluation  criteria. 

a.  Derive  importance  weights  for  each  factor  identified 
in  Step  2-b. 

b.  Assess  utility  of  each  operational  measure  (the 
output  of  phase  3). 

6.  Generate  systems-versus-criteria  array. 

a.  Obtain  system  effectiveness  ratings. 

b.  Determine  partial  utility  scores  for  each  factor. 

c.  Aggregate  partial  utilities  to  produce  a  global 
utility  score  for  each  alternative. 

d.  Estimate  cost  for  each  alternative. 

7.  Perform  sensitivity  analyses. 

8.  Select  a  preferred  alternative. 
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Since  the  CIEA  is  a  demonstration  exercise,  the  needs/constraints  assess¬ 
ment  portion  (Phase  1)  was  not  formally  addressed.  For  purposes  of  analy¬ 
sis,  the  following  assumptions  were  made:  (1)  concern  has  been  expressed 
regarding  the  current  level  of  rifle  marksmanship  proficiency  in  the  Army; 

(2)  leadership  concern  has  been  formalized  in  a  problem  statement;  and 

(3)  no  relevant  constraints  have  been  specified.  Results  from  each  of  the 
remaining  phases  of  the  demonstration  analysis  are  now  presented  and  dis¬ 
cussed  in  turn. 


Demonstration  CIEA 


Define  System  Objectives 

The  objective  of  Phase  2,  Define  System  Objectives,  is  to  translate 
the  general  problem  statement  into  a  formal  specification  of  the  objec¬ 
tives  of  the  proposed  DORAC.  This  is  accomplished  through  a  process  con¬ 
sisting  of  three  steps,  listed  as  follows: 

1.  Determine  the  issues  to  which  DORAC  information  is  relevant. 

2.  Determine  information  worth  dimensions. 

3.  Identify  the  perspective  from  which  information  worth 
is  to  be  assessed. 

Again  based  on  assumption,  a  list  of  information  issues  relevant  to  marks¬ 
manship  proficiency  is  presented  in  Table  4-1.  The  issues  listed  in 
Table  4-1  represent  the  starting  point  for  the  demonstration  CIEA. 

The  information  issues  presented  in  Table  4-1  primarily  concern  opera- 
nal  readiness.  An  additional  area  of  interest  is  unit  training  man- 
z  lent.  Hence,  the  information  worth  dimensions  (WDs)  selected  for  the 
demonstration  CIEA  are  listed  and  defined  as  follows: 

1.  Readiness  Evaluation.  The  determination  of  whether  or  not 
individuals/units  are  capable  of  performance  at  an  accept¬ 
able  level/standard  on  the  performances  specific  to  the 
DORAC  application. 

2.  Training  Management.  The  use  of  performance  status  or 
diagnostic  information  in  determining  who,  how  often,  when, 
and  what  to  train  for  individual/unit  performances  related 
to  the  specific  DORAC. 

Other  potential  information  applications  such  as  unit  management  or  fight¬ 
ing  system  evaluation/development  were  not  considered  in  the  demonstration 
analysis. 
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Table  4-1 


Information  Issues 


1.  Can  soldiers  prepare  the  M16A1  rifle  for  operation? 

2.  Can  soldiers  maintain  the  M16A1  rifle? 

3.  Can  soldiers  fire  on  probable  targets  from  positions  dictated 
by  combat  and  under  combat  conditions? 

4.  Can  soldiers  detect  and  identify  the  types  of  targets  probable 
in  combat? 

5.  Can  soldiers  be  used  effectively  in  assault  operations? 

6.  Can  soldiers  be  used  effectively  in  defense  operations? 

7.  Can  soldiers  be  expected  to  practice  ammunition  conservation? 

8.  Can  soldiers  be  expected  to  engage  hostile  targets  presenting 

the  greatest  threat? 

9.  Can  soldiers  be  expected  to  maintain  proper  concealment  under 
combat  conditions? 

10.  Can  fire  teams  be  expected  to  maintain  adequate  weapon  operation, 
and  engagement  procedures  to  enhance  the  probability  that  they  can 
effectively  operate  in  combat? 


After  defining  information  issues  and  WDs,  the’final'  step  in  Phase  2 
involved  specifying  the  perspective  from  which  information,  worth  was  to 
be  assessed.  As  noted  in  Hawley  and  Dawdy  (1981),  the  issue  of  "utility 
to  whom"  is  decided  by  referring  to  the  WDs.  In  the  present  situation, 
the  WDs  indicate  that  subject  matter  experts  in  the  areas  of  readiness 
evaluation,  training  evaluation,  and  unit  training  represent  the  necessary 
perspective.  Within  the  Army,  such  expertise  is  found  in  personnel  from 
affected  operational  units  and  from  the  proponent  school  for  the  parent 
materiel  system.  Accordingly,  a  decision  was  made  to  conduct  the  demon¬ 
stration  analysis  using  representatives  from  an  Infantry  unit  and  the  U.S. 
Army  Infantry  School  Directorate  of  Training  Development  (USAIS  DTD). 


Identify  Operational  Requirements 

The  next  phase  in  CIEA  was  to  translate  the  information  issues  into 
operational  requirements  for  a  DQRAC;  that  is,  to  map  information  issues 
into  specific  individual /unit  performances.  In  addition,  the  conditions 
under  which  performances  are  to  be  demonstrated  and  the  standards  by  which 
performances  are  evaluated  were  also  specified.  Table  4-2  presents  a  list 
of  the  performances  and  standards  specifically  associated  with  Readiness 
Evaluation.  Table  4-3  lists  the  performances  and  standards  relevant  to 
training  management.  Potential  performance  conditions  for  both  sets  of 
performances  are  given  in  Table  4-4. 


Develop  Alternative  System  Concepts 

As  noted  previously,  the  training  devices  selected  for  use  in  the 
demonstration  CIEA  are  relevant  to  the  issue  of  marksmanship  proficiency. 
The  current  method  used  to  assess  marksmanship  proficiency  is  RF  conducted 
one  time  per  year.  During  RF,  each  soldier  is  taken  to  a  firing  range 

and  assessed  in  a  40-round  live-fire  exercise.  Prone  and  foxhole  firing 

positions  are  employed;  range  and  target  exposure  time  also  vary.  A 
candidate  is  rated  as  "qualified"  if  he/she  achieves  17  (23  at  Ft. 

Benning)  or  more  hits  out  of  40  possible.  Figure  4-1  presents  the  firing 
positions,  target  ranges,  and  times  used  in  RF. 

Weaponeer  (WP)  is  an  M16A1  remedial  marksmanship  trainer  designed  to 
isolate  individual  performance  deficiencies.  A  simulated  M16A1  rifle  is 
equipped  with  a  target  sensor  and  each  target  contains  a  light  emitting 
diode  (LED)  which  is  sensed  by  the  target  sensor  on  the  rifle.  A  predicted 
round  impact  point  is  determined  by  the  LED-target  sensor  alinement.  WP 

has  a  memory  for  recording  up  to  32  predicted  shot  impacts  and  a  printer 

for  providing  a  printout  of  all  shots  on  selected  targets.  Rifle  recoil 
is  simulated  with  recoil  energy  being  variable  from  no  recoil  to  a  recoil 
intensity  40  percent  greater  than  the  recoil  of  a  standard  M16A1  rifle. 
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Table  4-2 


Performances  and  Standards  Relevant  to  Readiness  Evaluation 


PERFORMANCE 

I.  Condition  to  operate 
A.  calibrate  (zero) 


II.  Maintain  operation 

A.  reload 

B.  reduce  a  stoppage 
III.  Operate 

A.  aimed  fire  from: 

1.  prone 

2.  sitting 

3.  squatting 

4 .  kneeling 

5.  standing 

6.  foxhole  supported 

7.  foxhole  unsupported 

B.  pointed  fire  from: 

1.  standing 

2.  crouching 

3 .  moving 

IV .  Engage 

A.  detect  and  identify 
targets 

B.  assault 


C.  defense 

D.  patrol  operation 

E.  ammunition  rationing 


STANDARD 


prescribed  shot  group  and  shot  group 
location 

proper  weapon  operation 


continued  weapon  operation 

return  the  weapon  to  proper  operation 

No  standard  established  by  position. 
Current  standard:  17/40  (42%)  on  record 
fire  collapsed  across  the  foxhole  and 
prone  position  and  confounded  with  target 
range  and  time  of  exposure.  The  initial 
standard  can  be  set  at  the  approximate  42% 
level  used  for  record  fire. 


No  standard  stated.  The  initial  standard 
can  be  a  minimum  at  42%  hits  on  hostile 
and  a  reduced  score  for  hitting  cn  a  friend. 

No  standard  stated.  The  initial  standard 
can  be  a  minimum  of  42%. 

No  standard  stated.  The  initial  standard 
can  be  a  minimum  of  42%. 

See  ’assault'  above. 

No  standard  stated.  The  initial  standard 
can  consist  of  allocating  a  number  of  rounds 
based  on  the  number  of  targets  presented 
plus  y%.  y%  based  on  prescribed  firing 
modes  for  targets  presented.  Require  the 
firer  to  fire  on  90%  of  the  targets  pre¬ 
sented  and  have  an  overall  accuracy  of  42%. 


4. 

At  Ft.  Benning  the  standard  for  qualification  is  23/40  or  58%. 
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Table  4-2  (Cont’d) 


PERFORMANCE 
XV.  Engage  (Cont’d) 

F.  threat  evaluation 

G.  concealment 


V.  Team  fire 


STANDARD 


No  standard  stated. 

Nc.  standard  stated.  The  initial  standard 
can  be  based  on  duration  of  exposure 
weighted  by  amount  of  exposure  and  target 
range. 

No  standard  stated. 

See  appropriate  comments  under  'maintain 
operation'  and  'engage'  above. 


Table  4-3 


Performances  and  Standards  Relevant  to  Training  Management 


PERFORMANCE  STANDARD 


1.  Condition  to  operate 
A.  Calibration  (zero) 


1. 

adjust  for  minute 
of  angle 

2. 

adjust  for  gravity 
effect 

3. 

sight-weapon  alinement 

B. 

Load 

1. 

inspect  magazine  top 
round  alinement 

2. 

apply  proper  inten¬ 
sity  blow  to 
seat  magazine 

3. 

insure  the  first  round 
is  seated  in  the 
chamber 

Maintain  operation 

A. 

Reload 

1. 

remove  the  magazine 

2. 

see  'load' 

B. 

Reduce  a  stoppage 

l. 

perform  immediate 
action  procedure 

2. 

inspect  an  M-16  and 
diagnose  the  stoppage 

3. 

take  the  appropriate 
action  for  reducing 
a  stoppage 

prescribed  shot  group  and  shot  group 
location 

prescribed  procedure 

prescribed  procedure 

prescribed  procedure 
proper  weapon  operatioir 
prescribed  procedure 

prescribed  procedure 

prescribed  procedure 

continued  weapon  operation 

prescribed  procedure 
see  'load' 

return  the  weapon  to  proper  operation 
prescribed  procedure 

prescribed  procedure 

identification  of  the  correct 
procedure  and  a  prescribed 
procedure 


4-8 


>  "ivy.*,  -v  • 


* 


Table  4-3  (Coat'd) 


PERFORMANCE 


III.  Operate 

A.  Mar-machine  Interface 

(M-MI) 

1.  trigger  control 

(squeeze) 

2 .  grip  on  weapon 

3.  breath  control 

4.  relax 

5.  compensate  for  recoil 

6.  change  firing  mode 

7.  stock  weld  (if 

applicable) 

8.  firing  elbow  (if 

applicable) 

B .  Marksmanship 

1.  compensate  for  range 


2.  compensate  for  wind 


3.  compensate  for  target 

movement 

4.  transfer  of  fire 

5.  obtain  a  proper  3ight 

picture  (aimed 
fire  only) 

6.  obtain  a  proper  aiming 

point  (aimed  fire 
only) 

7.  estimate  an  adequate 

weapon  aspect,  round 
impact  point  rela¬ 
tionship  (pointed 
fire) 


STANDARD 


prescribed  technique 


No  standard  established  by  range.  The 
initial  standard  can  be  a  minimum  of 
42%  hits. 

No  standard  established  by  wind  velocity. 
The  initial  standard  can  be  a  minimum 
of  42%  hits  under  wind  conditions. 

No  standard  established.  The  initial 
standard  can  be  a  minimum  of  42%  hits 
on  moving  targets. 

No  standard  established.  The  initial 
standard  can  be  a  minimum  of  42%  hits. 

prescribed  procedure 


prescribed  procedure 


No  standard  established  for  pointed  fire. 
Since  pointed  fire  is  appropriate  only 
for  targets  at  close  range,  an  initial 
standard  of  40%  hits  can  be  used. 
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PERFORMANCE 
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Table  4-3  (Cont'd) 


STANDARD 


III.  Operate  (Cont'd) 

B.  Marksmanship  (Cont'd) 

8.  aimed  fire  from: 

a.  prone 

b.  sitting 

c.  squatting 

d.  kneeling 

e.  standing 

f.  foxhole  supported 

g.  foxhole  unsupported 


No  standard  established  by  position. 
Current  standard:  17/40  on  record  fire 
collapsed  across  the  foxhole  and  prone 
position  and  confounded  with  target  range 
and  time  of  exposure.  The  initial 
standard  can  be  set  at  the  approximate 
42%  level  used  for  record  fire. 


9.  pointed  fire  from: 

a.  standing 

b .  crouching 

c .  moving 

IV .  Engage 

A.  Detect,  and  identify 

targets 

B.  Assault 

C.  Defense 

D.  Patrol  operation 

E.  Ammunition  rationing 


F.  Threat  evaluation 

G.  Concealment 


No  standard  stated.  The  initial  standard 
can  be  a  minimum  at  42%  hits  on  hostile 
with  a  reduced  score  for  hitting  a  friend. 

No  standard  stated.  The  initial  standard 
can  be  a  minimum  of  42%. 

No  standard  stated.  The  initial  standard 
can  be  a  minimum  of  42%. 

See  'assault'  above. 

No  standard  stated.  The  initial  standard 
can  be  allocating  a  number  of  rounds  based 
on  the  number  of  targets  presented  plus 
y%.  y%  based  on  prescribed  firing  modes 
for  targets  presented.  Require  the  firer 
to  fire  on  90%  of  the  targets  presented 
and  have  an  overall  accuracy  of  42%. 

No  standard  stated. 

No  standard  stated.  The  initial  standard 
can  be  based  on  duration  of  exposure 
weighted  by  amount  of  exposure  and  target 
range . 


V.  Team  Fire 


No  standard  stated. 


See  appropriate  comments  under  'maintain 
operation*  and  'engage*  above. 
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Table  4-4 


Potential  Performance  Conditions 


1.  Motion  (firer  movement) 

2.  Illumination 

3.  Meteorological 

4.  Clothing  worn  by  firer 

5.  Temperature 

6.  Target-background  contrast  ratio 

7.  Similarity  between  friend  and  enemy 

8.  Terrain 

9.  Field  of  fire 

10.  Approach  routes 

11.  Concealment  factors 

12.  Force  density  ratio 

13.  Fire  sector 

14.  Firer-target  proximity 
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Three  types  of  magazines  are  provided  for  use  with  the  rifle:  a  20-round 
(unlimited  fire)  magazine,  a  30-round  (unlimited  fire)  magazine,  and  a 
limited  fire  magazine  that  allows  from  1  to  30  simulated  rounds  in  the 
magazine.  A  headset  is  provided  for  simulating  the  firing  sound  of  an 
M16A1  rifle.  The  WP  also  includes  a  selection  for  random  misfire. 

WP  can  present  three  targets:  a  scaled  25  meter  zeroing  target,  a 
scaled  100  mete""  'E'  type  silhouette  target,  and  a  250  meter  'E'  type 
silhouette  target.  Any  target  selected  can  be  raised  at  random  during  a 
1  to  9  second  time  frame  and  can  remain  in  a  raised  position  for  a  dura¬ 
tion  of  2,  3,  5,  10,  15,  20,  25  seconds,  or  continuous.  The  WP  provides 
a  target  'Kill'  function:  a  selection  that  will  cause  a  raised  target  to 
drop  when  it  is  hit.  Firing  pads  used  with  the  WP  provide  the  capability 
for  the  firer  to  fire  from  the  foxhole  or  prone  position. 

A  video  display  allows  an  observer  to  monitor  individual  shots  and 
replay  the  last  3  seconds  of  each  of  the  first  3  shots.  Scoring  available 
with  the  WP  video  display  includes:  the  target  on  display,  the  number  of 
hits  on  the  target,  the  number  of  misses,  late  shots  (fired  after  target 
drops),  the  shot  number,  and  the  total  number  of  shots  fired  (Spartanics, 
Inc. ,  1976) . 

The  second  training  device  was  the  SWAT.  SWAT  is  an  electrooptics- 
based,  microcomputer-controlled,  training  device  that  permits  tactical 
infantry  weapons  training  under  stimulated  battlefield  conditions  in  a 
classroom  environment.  In  a  short  period  of  time,  a  user  can  be  subjected 
to  a  wide  range  of  combat  situations  where  each  user's  performance  is 
analyzed  in  real-time,  and  immediate  feedback  is  given  to  both  the  users 
and  the  evaluator.  Combat  scenarios  can  be  changed  to  fit  any  potential 
battlefield  requirement.  The  trainer  is  configured  for  a  maximum  of  five 
persons . 

The  SWAT  has  two  motion  picture  projectors:  a  visual  and  an  infrared 
(IR)  projector.  The  visual  projector  displays  the  battle  scene,  including 
visual  targets.  The  infrared  projector  provides  invisible  infrared  target 
areas  which  the  weapon  must  be  aimed  at  in  order  to  score  a  hit.  Lead  is 
programmed  into  the  infrared  target,  which  the  weapon  receiver  detects, 
requiring  the  trainee  to  lead  the  target  as  necessary.  Each  user  has  a 
simulated  M16A1  rifle  with  an  attached  IR  detector  consisting  of  a  four- 
quadrant  photo  diode.  The  four-quadrant  target  information  and  micro¬ 
computer  logic  determine  kills,  eight  areas  of  near  misses,  and  total 
misses . 

When  the  user  fires  the  weapon,  he/she  hears  a  simulated  bang  and 
feels  a  recoil.  Recoil  is  generated  by  a  short  pulse  of  air  released  near 
the  front  sight  which  drives  the  weapon  high  and  to  the  right.  An  8080- 
based  micro-processor  determines  where  the  round  would  have  hit  and  sup¬ 
plies  this  information  to  both  a  computer-generated  voice  unit  and  a 
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cathode-ray  tube  (CRT)  display  on  the  evaluator's  station.  The  computer 
voice  unit  drives  both  the  user's  and  evaluator's  headsets.  When  a  target 
appears  on  the  screen,  the  IR  projector  outputs  a  target  present  signal. 
This  signal  starts  a  clock  in  the  microcomputer  which  measures  the  time 
until  the  user  fires,  thus  assessing  reaction  time.  The  target  present 
signal  is  also  used  to  determine:  (1)  the  number  of  targets  that  appeared, 
(2)  targets  ignored,  (3)  targets  shot  at,  and  (4)  if  the  user  fired  when 
no  target  was  present.  User  results  are  continuously  displayed  in  five 
columns  on  a  CRT  display  at  the  evaluator's  station.  At  the  completion  of 
the  exercise  the  results,  analyses  and  response  time  are  printed  by  a 
terminal  at  the  evaluator's  station. 

Distribution  cf  fire  is  monitored  using  an  IR  source  located  in  the 
flash-hider  part  of  the  rifle.  The  projected  IR  laser  spot  is  invisible 
to  the  user  but  is  detected  by  an  infrared  television  camera  and  displayed 
by  a  CRT  located  on  the  evaluator's  console.  When  the  rifle  is  fired,  the 
IR  spot  projector  illuminates  the  CRT  with  a  small  IR  aim  spot.  If  the 
evaluator  wants  to  continuously  monitor  rifle  motion,  the  IR  aiming  spot 
can  be  left  on  continuously.  Data  can  also  be  recorded  for  playback  dur¬ 
ing  debriefing.  A  more  complete  description  of  SWAT  is  provided  in 
Andrews  (1979). 

Training  devices  alone  will  not  always  constitute  DORAC  alternatives. 
In  fact,  DORAC  alternatives  will  usually  consist  of  sets  of  training 
devices/methods  used  in  combination  and  a  usage  scenario.  In  the  CIEA 
demonstration,  the  following  devices/methods  and  usage  scenarios  consti¬ 
tuted  the  DORAC  alternatives: 

1.  Baseline.  The  present  situation  of  RF  conducted  one 
time  per  year.  [RF(1)] 

2.  RF  twice  per  year  (every  six  months).  [RF(2)j 

3.  RF  quarterly.  [RF(4)] 

4.  WP  once  per  year.  [WP(1)] 

5.  RF  once,  WP  once.  [RF(1)  +  WP(1)] 

6.  RF  once,  WP  three  times.  [RF(1)  +WP(3)] 

7.  SWAT  once  per  year.  [SWAT(l)] 

8.  RF  once,  SWAT  once.  [RF(1)  +  SWAT(l)] 

9.  RF  once,  SWAT  three  times.  [RF(1)  +  SWAT(3)] 

10.  RF,  WP,  and  SWAT  used  in  a  rotating  quarterly  evaluation. 

[RF  +  WP  +  SWAT] 
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The  alternatives  were  reviewed  to  identify  which  of  the  performances 
identified  in  Tables  4-2  and  4-3  could  be  assessed  by  each  alternative. 
Table  4-5  presents  the  resulting  Performance  by  Alternative  Matrix  for 
Training  Management.  Readiness  Evaluation  performance  measures  constitute 
a  subset  of  the  Training  Management  measures.  These  measures  are  shown 
separately  in  Table  4-6. 

Because  of  the  nature  of  the  conditions  for  some  of  the  M16A1  perform¬ 
ances  (i.e.,  many  conditions  are,  in  fact,  "catch  alls"  that  refer  to  a 
broad  range  of  possible  situations),  a  Performance  by  Conditions  Matrix 
as  discussed  in  Hawley  and  Dawdy  (1981)  was  not  developed.  Rather,  a  de¬ 
scription  of  the  measurement  potential  of  each  of  the  training  devices  in 
terms  of  performance  conditions  and  target  variables  is  provided  in  Table 
4-7. 


Finally,  Table  4-8  presents  measurement  method  descriptions  and  judged 
data  dependability  ratings  for  each  device/method  on  each  performance. 
Information  quality  ratings  were  assigned  using  the  5-point  scale  given 
as  follows: 

1  -  Very  Low 

2  -  Low 

3  -  Moderate 

4  -  High 

5  -  Very  High 


Establish  System  Evaluation  Criteria 

The  next  step  in  the  CIEA  demonstration  involved:  (1)  deriving  im¬ 
portance  weights  for  WDs,  and  (2)  obtaining  utility  ratings  for  opera¬ 
tional  measures  (OMs)  (i.e..  Performances)  nested  within  WDs.  In  the 
demonstration  exercise,  these  sets  of  parameters  were  obtained  using  two 
independent  groups  of  decision-makers. 

Since  the  WDs  of  interest  were  Readiness  Evaluation  and  Training 
Management,  the  proper  perspective  for  information  worth  assessment  was 
unit  readiness  assessment,  training  development,  training  evaluation,  and 
unit  training.  Accordingly,  importance  weights  and  utility  ratings  were 
elicited  in  a  collective  setting  using  independent  groups  consisting  of 
personnel  from  1/58  Infantry  at  Ft.  Benning,  GA,  and  from  USAIS  DTD,  In 
both  cases,  consensus  judgments  were  obtained. 

The  first  task  for  the  two  groups  of  raters  was  to  provide  importance 
weights  for  the  two  WDs.  Separate  results  from  the  two  groups  and  averaged 
weights  are  presented  as  follows: 
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Table  4-5 


Performance  by  Alternative  Matrix  for  Training  Management1 


Individual  1 

Device 

Devices 

Combinations 

H 

* 

t n 

£ 

4* 

% 

& 

H 

+ 

+ 

+ 

p- 

& 

U- 

tn 

Ou 

02 

s 

V) 

02 

cc 

02 

Performance 

I. 

Condition  to  operate 

A. 

Calibration  (zero) 

1.  adjust  for  minute  of  angle 

X 

0 

0 

X 

X 

X 

2.  adjust  for  gravity  effect 

X 

0 

0 

X 

X 

X 

3.  sight  weapon  alinement 

X 

0 

0 

X 

X 

X 

B. 

Load 

1.  inspect  top  round  alinement 

X 

0 

0 

X 

X 

X 

2.  apply  blow  to  seat  magazine 

X 

X 

X 

X 

X 

X 

3.  insure  the  first  round  is 

X 

0 

0 

X 

X 

X 

seated  in  the  chamber 

II. 

Maintain  operation 

A. 

Reload 

1.  remove  the  magazine 

X 

X 

X 

X 

X 

X 

2.  see  'load' 

X 

0 

0 

X 

X 

X 

B. 

Reduce  a  stoppage 

1.  perform  immediate  action 

X 

X 

X 

X 

X 

X 

procedure 

2.  Inspect  an  M-16  and  diagnose 

X 

0 

0 

X 

X 

X 

the  stoppage 

3.  take  the  appropriate  action 

X 

0 

0 

X 

X 

X 

for  reducing  a  stoppage 

1 

1 Table  entries  are  an  "X"  or  an  "0"  indicating  that  the  device/device  com¬ 
bination  does  or  does  not,  respectively,  provide  a  vehicle  for  assessing 
a  given  performance. 
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Table  4-5  (Cont'd) 


Performance 
III.  Operate 

A.  Man-machine  interface 

1.  trigger  control 

2.  grip  on  weapon 

3.  breath  control 

4.  relax 

5.  compensate  for  recoil 

6.  change  firing  mode 

7.  stock  weld 

8.  firing  elbow 
B.  Marksmanship 

apensate  for  range 

2.  compensate  for  wind 

3.  compensate  for  target  movement 

4.  transfer  of  fire 

5.  obtain  a  proper  sight 

picture  (aimed  fire) 

6.  obtain  a  proper  aiming  point 

(aimeH  cire) 

7.  estimr  ....  adequate  weapon 

aspect,  round  impact  point 
relationship  (pointed  fire) 
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RF  +  SWAT 


Table  4-5  (Cont’d) 


1 

Performance 

Individual 

Devices 

H 

< 

n.  Pj 

a:  5  vj 

Device 

Combinations 

H 

< 

<  + 

&  3  £ 

+  +  + 

fa  fa  fa 

OS  pci  prf 

III. 

Operate  (cont'd) 

B.  Marksmanship  (cont’d) 

8.  aimed  fire  from: 

a.  prone 

X 

X 

X 

X 

X 

X 

b.  sitting 

X 

X 

X 

X 

X 

X 

c.  squatting 

X 

X 

X 

X 

X 

X 

d.  kneeling 

X 

X 

X 

X 

X 

X 

e.  standing 

X 

X 

X 

X 

X 

X 

f.  foxhole  supported 

X 

X 

X 

X 

X 

X 

g.  foxhole  unsupported 

X 

X 

X 

X 

X 

X 

9.  pointed  fire  from: 

a.  standing 

X 

0 

X 

X 

X 

X 

b.  crouching 

X 

0 

X 

X 

X 

X 

c .  moving 

0 

0 

0 

0 

0 

0 

IV. 

Engage 

A.  Detect  and  identify  targets 

0 

0 

X 

0 

X 

0 

B.  Assault 

0 

0 

0 

0 

0 

0 

C.  Defense 

0 

0 

X 

0 

X 

X 

D.  Patrol  operation 

0 

0 

0 

0 

0 

0 

E.  Ammunition  rationing 

0 

0 

0 

0 

0 

X 

F.  Threat  evaluation 

0 

0 

0 

0 

0 

0 

G.  Concealment 

0 

0 

0 

0 

0 

0 

V. 

Team  fire 

0 

0 

X 

0 

X 

X 
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V.  Team  fire 


Table  4-6 


Performance  by  Alternative  Matrix  for  Readiness  Evaluation 


Performance 

I.  Condition  to  operate 
A.  Calibrate  (zero) 

B •  Load 

II.  Maintain  operation 

A.  Reload 

B.  Reduce  a  stoppage 
III.  Operate 

A.  Aimed  fire  from: 

1 .  prone 

2.  sitting 

3.  squatting 

4.  kneeling 

5.  standing 

6.  foxhole  supported 

7.  foxhole  unsupported 

B.  Pointed  fire  from: 

1.  standing 

2.  crouching 

3 .  moving 


Individual 

Devices 


H 

< 

u.  p-  be 

a  s  w 


X  0  0 

X  0  0 

0  0  0 

0  0  0 


XXX 
XXX 
XXX 
XXX 
XXX 
X  X  0 

X  X  0 

XXX 
XXX 
0  0  0 


Device 

Combinations 

H 

&  I 

+  + 


XXX 

XXX 

0  0  0 

0  0  0 


XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 
XXX 
0  0  0 
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RF  +  WP  +  SWAT 


Table  4-6  (Cont'd) 


Performance 
IV.  Engage 

A.  Detect  and  identify  targets 

B.  Assault 

C.  Defense 

D.  Patrol 

E.  Ammunition  rationing 

F.  Threat  evaluation 

G.  Concealment 
V.  Team  fire 


Individual  Device 

Devices  Combinations 


H 

31 

in 


+ 

Uj 

OC 


H 

< 

<n 


ai 


0  0  X 
0  0  0 
0  0  X 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  X 


0X0 
0  0  0 
0  X  X 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 


RF  +  WP  +  SWAT 


Table  4-7 


Training  Device  Capabilities  in  Terms  of 
Performance  Conditions  and  Target  Variables 


Target  Variables: 

1.  Multiple 

2.  Friendly  and  hostile 

3.  Range  variable 

4 .  Movement 

A.  Direction 

B.  Distance 

C.  Rate 

5.  Exposure 

A.  Amount 

B.  Duration 

C.  Frequency 

6.  Camouflage 

7.  Target  termination  when  hit 
Conditions : 

1.  Motion  (firer  movement) 

2.  Illumination 

3.  Meteorological 

4.  Clothing  worn  by  firer 

5.  Temperature 

6.  Target-background  contrast  ratio 

7.  Similarity  between  friend  and  enemy 

8.  Terrain 

9.  Field  of  fire 


Device/Method 

&  & 


0  0  X 

0  0  X 

X  0  X 

o  o  x 

0  0  X 

0  0  X 

0  0  X 

XXX 
XXX 
0  0  X 

X  X  o 

0  0  0 

XXX 
0  0  0 

XXX 
0  0  0 

0  0  X 

0  0  X 

0  0  X 

0  0  X 
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SWAT 


r 


Table  4-7  (Cont’d) 


Conditions  (Cont’d) 

10.  Approach  routes 

11.  Concealment  facts 

12.  Force  density  ratio 

13.  Fire  sector 

14.  Firer-target  proximity 


Device/Method 

&  £  35 


0  0  X 
0  0  0 
0  0  0 
0  0  0 
0  0  X 


I 


Table  4-8 

Measurement  Method  Descriptions  and  Dependability 2  Ratings 
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2Dependability  ratings  for  human  evaluators  assume  ideal  conditions. 


Table  4-8  (Cont’d) 


standing  score  by  position  and  firer's  score  by 

crouching  composite.  Currently  position.  Firer 

moving  not  assessed.  movement  not  possible 


Worth  Dimension 


1/58  Infantry  USAIS  DTD 


Combined 


Readiness  Evaluation  .80  .83  .815 

Training  Management  .20  .17  .185 

The  judged  importance  weights  were  quite  consistent  across  the  two  groups 
of  raters. 

After  assigning  importance  weights  to  the  WDs ,  each  group  next  assigned 
utility  ratings  to  operational  OMs  nested  within  WDs.  Each  of  the  groups 
was  assigned  to  rate  one  set  of  OMs:  personnel  from  1/58  Infantry  rated 
the  measures  nested  under  Operational  Readiness;  the  USAIS  DTD  participants 
rated  the  measures  nested  under  Training  Management. 

The  final  utility  ratings  for  Readiness  Evaluation  are  presented  in 
Table  4-9.  Similar  results  for  the  measures  nested  under  Training  Manage¬ 
ment  are  listed  in  Table  4-10. 


Generate  Systems-Versus-Crlteria  Array 

Following  the  generation  of  importance  weights  for  WDs  and  utility 
scores  for  OMs  nested  within  WDs,  the  next  phase  in  MAUM-CIEA  involved 
generating  the  systems-versus-criteria  array.  In  CIEA,  the  systems-versus- 
criteria  array  is  a  matrix  that  presents  each  alternative  and  its  asso¬ 
ciated  evaluation  criteria — Information  Utility,  Relative  Information 
Utility,  Information  Cost,  Relative  Information  Cost,  and  Relative  In¬ 
formation  Worth. 

Developing  the  systems-versus-criteria  array  was  a  five  step  pro¬ 
cedure,  with  the  steps  listed  as  follows: 

1.  Obtain  System  Effectiveness  Ratings 

2.  Determine  Partial  Utilities  for  WDs 

3.  Aggregate  Partial  Utility  Scores 

4.  Estimate  Costs  of  Alternatives 

5.  Determine  Relative  Information  Worth 

Each  of  these  steps  is  discussed  in  the  following  paragraphs. 

Obtain  Systems  Effectiveness  Ratings.  Within  the  context  of  CIEA, 
system  effectiveness  is  defined  as  the  degree  to  which  an  alternative  pro¬ 
vides  timely,  quality  information  on  the  performances  relevant  to  the 
DORAC  implementation.  Specifying  system  effectiveness  was  carried  out 
in  three  substeps:  First,  information  quality  ratings  were  obtained  for 
each  device/device  combination  (d/dc)  on  each  OM.  Next,  each  d/dc  was 
evaluated  with  respect  to  the  utility  of  the  frequency  with  which  per¬ 
formance  data  are  provided.  Finally,  the  quality  ratings  and  frequency 
utility  scores  were  combined  to  produce  a  single  measure  of  the  effective¬ 
ness  of  each  d/dc  on  each  OM  nested  within  each  WD. 
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Table  4-9 


Utility  Ratings  for  Information  Measures 
Nested  Under  Readiness  Evaluation 


I. 


II, 


III. 


IV. 


V. 


Condition  to  operate 


A. 

Calibrate  (zero) 

.0265 

B. 

Load 

.0305 

Maintain  operation 

A. 

Reload 

.0287 

B. 

Reduce  a  stoppage 

.0269 

Operate 

A. 

Aimed  fire  from: 

1.  prone 

.0896 

2,  sitting 

.0108 

3.  squatting 

.0086 

4.  kneeling 

.0090 

5 .  standing 

.0262 

6.  foxhole  supported 

.0802 

7.  foxhole  unsupported 

.0806 

B. 

Pointed  fire  from: 

1.  standing 

.0179 

2 .  crouching 

.0609 

3 .  moving 

.0627 

Engage 

A. 

Detect  and  identify  targets 

.1075 

B. 

Assault 

.0717 

C. 

Defense 

.0860 

D. 

Patrol 

.0538 

E. 

Ammunition  rationing 

.0079 

F. 

Threat  evaluation 

.0358 

G. 

Concealment 

.0082 

Team  fire 

.0699 

n 
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Table  4-10 


Utility  Ratings  for  Information  Measures 
Nested  Under  Training  Management 


I.  Condition  to  operate  (zero) 
A.  Calibration  (zero) 


...  Just  for  minute  of  angle 

.0034 

adjust  for  gravity  effect 

.0062 

3. 

sight  weapon  allnement 

.0103 

B. 

Load 

1. 

inspect  top  round  allnement 

.0041 

2. 

apply  blow  to  seat  magazine 

.0027 

3. 

insure  the  first  round  is 
seated  in  the  chamber 

.0027 

Maintain  operation 

A. 

Reload 

1. 

remove  the  magazine 

.0025 

2. 

see  ’load' 

B. 

Reduce  a  stoppage 

1. 

perform  Immediate  action  procedure 

.0027 

2. 

inspect  an  M-16  and  diagnose  the  stoppage 

.0025 

3. 

take  the  appropriate  action  for  reducing 
a  stoppage 

.0025 

Operate 

A. 

Man-machine  interface 

1. 

trigger  control 

.0274 

2. 

grip  on  weapon 

.0024 

3. 

br&ath  control 

.0077 

4. 

relax 

.0038 

5. 

compensate  for  recoil 

.0001 

6. 

change  firing  mode 

.0003 

7. 

stock  weld 

.0024 

8. 

firing  elbow 

.0022 
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Table  4-10  (Cont'd) 


111.  Operate  (Cont’d) 


B. 

Marksmanship 

1.  compensate  for  range 

.0069 

2.  compensate  for  wind 

.0052 

3.  compensate  for  target  movement 

.0021 

4.  transfer  of  fire 

.0007 

5.  obtain  a  proper  sight  picture  (aimed  fire) 

.0048 

6.  obtain  a  proper  aiming  point  (aimed  fire) 

.0005 

7.  estimate  an  adequate  weapon  aspect,  round 

impact  point  relationship  (pointed  fire) 

8.  aimed  fire  from: 

« Quo- 

a.  prone 

.0110 

b.  sitting 

.0080 

c.  squatting 

.0008 

d.  kneeling 

.0082 

e.  standing 

.0016 

f.  foxhole  supported 

.0165 

g.  foxhole  unsupported 

9.  pointed  fire  from: 

.0137 

a.  standing 

.0004 

b.  crouching 

.0013 

c .  moving 

.0021 

Engage 

A. 

Detect  and  identify  targets 

.0096 

B. 

Assault 

.2469 

C. 

Defense 

.2743 

D. 

Patrol  operation 

.0007 

E. 

Ammunition  rationing 

.0026 

F. 

Threat  evaluation 

.0055 

G. 

Concealment 

.0023 

Team  fire 

.2880 
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Information  quality  is  defined  as  the  extent  to  which  a  d/dc  provides 
trustworthy  information  relevant  to  a  particular  OM.  Also  considered  as 
part  of  information  quality  was  the  amount  of  information  provided;  that 
is,  the  number  of  relevant  performance  conditions  that  are  addressed  by 
a  d/dc.  In  the  demonstration  CIEA,  information  quality  ratings  were  ob¬ 
tained  using  the  procedure  outlined  as  follows: 

1.  Order  the  d/dcs  from  "best"  to  "worst"  according  to 
the  degree  to  which  each  is  capable  of  providing 
quality  information  relevant  to  the  OM  under  con¬ 
sideration.  Raters  were  instructed  to  consider  the 
following  characteristics  in  making  quality  judgments: 

a.  Amount  of  .V.tc. ’.  at  ion.  The  number  of  relevant 
performance  ■: '.unlit ions  addressed. 

b.  Dependability.  The  judged  trustworthiness  of  the 
data.  This  was  obtained  from  the  dependability 

. ratings  made  previously  (i.  e. ,  Table  4-8). 

Ties  were  permitted.  If  one  or  more  of  the  alternatives 
were  judged  equivalent  in  terms  of  the  quality  of  the 
information  they  provided,  they  were  assigned  the  same  rank. 

2.  Numerically  position  the  best  and  worst  d/dcs  on  a  0-to-100 
scale.  The  following  benchmark  ratings  were  used  as  a 
guide: 

0  -  The  d/dc  provides  no  data  relevant  to  the 
OM  under  consideration. 

25  -  Marginal.  The  d/dc  provides  partial  data  on  the 
OM  and  the  data  are  likely  to  be  undependable 
(e.g.,  the  recording/scoring  method  is  poor  re¬ 
sulting  in  low  validity  or  low  reliability) . 

50  -  Adequate.  The  d/dc  provides  the  required  data 
but  some  dependability  problems  are  apparent. 

For  example,  the  most  appropriate  recording/scoring 
method  is  not  used  or  the  data  are  likely  to  have 
only  moderate  reliability. 

75  -  Good.  The  d/dc  provides  required  data  in  an 

acceptable  manner.  Recording  methods  are  accept¬ 
able;  reliability  is  likely  to  be  reasonably 
high. 
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100  -  Excellent.  The  d/dc  is  the  best  possible ,  given 

the  current  technical  state  of  the  art.  Recording 
methods  are  automated  and  precise;  reliability 
is  likely  to  be  very  high. 


3.  Position  the  remaining  alternatives  between  the  best  and 
worst  cases  on  the  0-to-100  scale.  Again,  the  benchmark 
rating  points  presented  previously  were  suggested  as  a  guide. 

Tables  4-11  and  4-12  present  the  d/dc  Quality  ratings  for  Readiness 
Evaluation  and  Training  Management,  respectively. 

After  information  quality  ratings  had  been  assigned  to  each  d/dc  on 
each  OM,  the  next  substep  in  obtaining  effectiveness  ratings  was  to  de¬ 
termine  the  utility  of  the  evaluation  frequency  associated  with  each  DORAC 
alternative.  Frequency  utility  ratings  were  obtained  by  applying  the 
following  actions  to  each  QM  for  each  alternative: 

1.  Consider  the  frequency  of  the  information  provided  by 
each  alternative  (e.g.,  quarterly,  twice  a  year,  yearly, 
etc.).  Now,  specifically  considering  the  highest  and 
lowest  frequencies,  rate  the  usefulness  of  receiving 
DORAC  generated  information  with  the  frequencies  indicated. 
Use  a  0-to-100  scale  in  assigning  the  ratings. 

2.  Next,  consider  the  remaining  (i.e.,  intermediate)  frequen¬ 
cies.  Position  the  remaining  frequencies  between  the 
extreme  values  (i.e.,  ratings  for  the  highest  and  lowest 
frequencies)  on  the  0-to-100  scale. 

The  results  of  the  frequency  utility  rating  procedure  are  presented 
in  Tables  4-13  and  4-14.  Zero  entries  indicate  that  an  alternative  did 
not  provide  a  vehicle  for  evaluating  a  given  performance. 

The  final  substep  in  the  process  of  obtaining  effectiveness  ratings 
for  alternatives  was  to  combine  the  quality  and  frequency  ratings  into  a 
single  measure  of  system  effectiveness.  Quality  and  frequency  were  com¬ 
bined  using  the  formula: 


where 


Eijk 

£ijk 


-  [Qijk  x  Fijk]/100,  (4-1) 

is  the  effectiveness  of  the  i*1*1  alternative  on  the 
k^  OM  nested  within  the  j*"*1  WD, 


is  the  Quality  for  the  training  devices  comprising 
the  i^  alternative  on  the  kt^  OM  nested  within 


and 


the  WD  (Tables  4-11  and  4-12),  ^ 

is  the  utility  of  obtaining  information  on  the  k 

til 

OM  nested  within  the  j  WD  with  the  frequency 
specified  by  the  it  alternative  (Tables  4-13  and  4-14). 
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Table  4-11 


Information  Quality  Ratings 
for  Training  Devices  and  Device  Combinations 
on  Readiness  Evaluation 


Individual 

Devices 


Device 

Combinations 


Condition  to  operate 

A.  Calibrate  (zero) 

B.  Load 

Maintain  operation 

A.  Reload 

B.  Reduce  a  stoppage 
Operate 

A.  Aimed  fire  from: 

1.  prone 

2.  sitting 

3.  squatting 

4.  kneeling 

5.  standing 

6.  foxhole  supported 

7.  foxhole  unsupported 

B.  Pointed  fire  from: 

1.  standing 

2 .  crouching 

3.  moving 

Engage 

A.  Detect  and  identify  targets 

B.  Assault 

C.  Defense 


pLt  Uri 

PL  PL 


75 

0 

75 

10 

10 

75 

10 

10 

75 

10 

10 

75 

50 

50 

75 

50 

50 

75 

50 

50 

75 

50 

50 

75 

50 

50 

75 

50 

50 

75 

50 

50 

0 

0 

50 

0 

0 

50 

El 

0 

0 

75 

0 

50 

75 

75 

75 

75 

75 

75 

75 

75 

75 

75 

75 

75 

75 

80 

80 

75 

80 

80 

75 

80 

80 

75 

80 

80 

75 

80 

80 

75 

80 

80 

75 

80 

80 

0 

50 

50 

0 

50 

50 

0 

0 

0 

75 

80 

80 

0 

0 

0 

0 

50 

50 

Engage  (Cont'd) 

D.  Patrol 

E.  Ammunition  rationing 

F.  Threat  evaluation 

G.  Concealment 


Device 

Combinations 


Team  fire 


0  40  40 

0  0  0 
0  25  25 


RF  +  WP  +  SWAT 


Table  4-12 


Information  Quality  Ratings 
for  Training  Devices  and  Device  Combinations 
on  Training  Management 


Individual 

1 

Device 

Devices 

Combinations 

H 

c 

5t 

Vi 

H 

+ 

£ 

S 

c n 

& 

+ 

+ 

+ 

U* 

St 

tK 

oc 

5 

C/3 

P5 

ed 

Condition 

to  operate 

A. 

Calibrate  (zero) 

1. 

adjust  for  minute  of  angle 

100 

0 

0 

100 

100 

100 

2. 

adjust  for  gravity  effect 

100 

0 

0 

100 

100 

100 

3. 

sight  weapon  alinement 

100 

0 

o 

100 

100 

100 

B. 

Load  I 

1. 

inspect  top  round  alinement 

100 

0 

90 

100 

100 

100 

2. 

apply  blow  to  seat  magazine 

100 

100 

100 

100 

100 

100 

3. 

insure  the  first  round 

100 

0 

75 

100 

100 

100 

Is  seated  in  the  chamber 

Maintain  o 

peration 

A. 

Reload 

1. 

remove  the  magazine 

100 

100 

100 

100 

100 

100 

2. 

see  'load* 

100 

0 

75 

100 

100 

100 

B. 

Reduce  a  stoppage 

1. 

perform  immediate  action 

100 

25 

90 

100 

100 

100 

procedure 

2. 

inspect  an  M-16  and  diagnose 

100 

0 

0 

100 

100 

100 

the  stoppage 

3. 

take  the  appropriate  action 

100 

0 

0 

100 

100 

100 

for  reducing  a  stoppage 

Operate 

A. 

Man-machine  interface 

1. 

trigger  control 

100 

90 

100 

100 

100 

100 
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Table  4-12  (Cont'd) 


Operate  (Cont’d) 

A.  Man-machine  interface  (Cont'd) 

2.  grip  on  weapon 

3.  breath  control 

4 .  relax 

5.  compensate  for  recoil 

6.  change  firing  mode 

7.  stock  weld 

8.  firing  elbow 

B .  Marksmanship 

1.  compensate  for  range 

2.  compensate  for  wind 

3.  compensate  for  target 

movement 

4.  transfer  of  fire 

5.  obtain  a  proper  sight 

picture  (aimed  fire) 

6.  obtain  a  proper  aiming 

point  (aimed  fire) 

7.  estimate  an  adequate  weapon 

aspect,  round  impact  point 
. .  —relationship  (pointed  fire) 


Individual 

Devices 

E-t 

■< 

&<  a 

«  3  71 

Device 

Combinations 

H 

§ 

<  + 

S  B5  g 

+  +  + 

U-I  tu  Pk 

os  pc 

100 

90 

100 

100 

100 

100 

100 

100 

90 

100 

100 

100 

100 

50 

75 

100 

100 

100 

100 

100 

75 

100 

100 

100 

25 

90 

100 

90 

100 

100 

100 

75 

90 

100 

100 

100 

100 

75 

90 

100 

100 

100 

90 

50 

0 

90 

90 

90 

90 

0 

0 

90 

90 

90 

0 

0 

90 

0 

90 

90 

90 

50 

100 

90 

100 

100 

100 

90 

100 

100 

100 

100 

90 

80 

100 

90 

100 

100 

60 

0 

0 

60 

60 

60 

Table  4-12  (Ccmt’d) 


Operate 

B. 

i 

(Cont ’ d) 

Marksmanship  (Cont'd) 

8.  aimed  fire  from: 

a.  standing 

Individual 

Devices 

H 

< 

tn  a,  ^ 

pi  5  W  : 

Device 

Combinations 

H 

< 

g 

+ 

&  g  g 

+  +  + 

u*  u* 

pc!  Ctf  Pd 

0 

90 

95 

90 

95 

95 

b.  sitting 

0 

90 

95 

90 

95 

95 

c.  squatting 

0 

90 

95 

90 

95 

95 

d.  kneeling 

0 

90 

95 

90 

95 

95 

e.  prone 

95 

90 

95 

95 

95 

95 

f.  foxhole  supported 

95 

90 

0 

95 

95 

95 

g.  foxhole  unsupported 

0 

90 

0 

90 

90 

90 

9.  pointed  fire  from: 

a.  standing 

0 

1 

90 

95 

90 

95 

95 

b .  crouching 

0 

90 

95 

90 

95 

95 

c •  moving 

o 

0 

25 

0 

25 

25 

Engage 

A. 

Detect  and  identify  targets 

10 

5 

90 

10 

90 

90 

B. 

Assault 

0 

0 

0 

0 

0 

0 

C. 

Defense 

10 

10 

50 

10 

50 

50 

D. 

Patrol  operation 

0 

0 

0 

0 

0 

0 

E. 

Ammunition  rationing 

0 

0 

75 

0 

0 

75 

F. 

Threat  evaluation 

0 

0 

80 

0 

0 

80 

G. 

Concealment 

0 

0 

0 

0 

0 

0 

Team  fire 

0 

0 

90 

0 

90 

90 

Table  4-13 


Frequency  Utility  Ratings  for  DORAC  Alternatives 
on  Readiness  Evaluation 


Condition  to  operate 

A.  Calibrate  (zero) 

B.  Load 

Maintain  operation 

A.  Reload 

B.  Reduce  a  stoppage 
Opercue 

A.  Aimed  fire  from: 

1.  prone 

2.  sitting 

3.  squatting 

4.  kneeling 

5.  standing 

6.  foxhole  supported  75 

7.  foxhole  75 

unsupported 


B.  Pointed  fire  from: 

1. 

standing 

0 

2. 

crouching 

0 

3. 

moving 

"0 

Engage 

A-  Detect  and  identify  75 
targets 

B.  Assault  0 


80  85  50 

80  85  50  75  80  50  80  85  88 


0 

0 

0 

0 

0 

50 

50 

60 

50 

0 

0 

0 

0 

0 

50 

50 

60 

50 

0 

0 

0 

0 

0 

0 

0 

0 

0 

80 

85 

0 

75 

75 

50 

80 

85 

82 

0 

0 

0 

0 

0 

0 

0 

0 

0 

05 


0- 

05 


U- 1 
at 


I— 1 

CO 

H 

/“\ 

V-/ 

< 

m 

H 

H 

& 

< 

< 

CO 

£ 

% 

& 

CO 

CO 

+ 

+ 

4* 

iH 

4* 

+ 

& 

/•"s 

/—•v 

s— ' 

✓“s 

« — 1 

I- 1 

rM 

H 

1— 1 

+ 

v-/ 

V 

v 

< 

•>*— / 

& 

P* 

12 

Pu, 

a* 

at 

at 

CO 

at 

05 

75 

80 

85 

0 

75 

75 

0 

75 

75 

75 

75 

75 

75 

10 

75 

75 

10 

75 

75 

75 

75 

75 

75 

10 

75 

75 

10 

75 

75 

75 

75 

75 

75 

10 

75 

75 

10 

75 

75 

75 

75 

80 

85 

50 

75 

80 

50 

80 

85  88 

75 

80 

85 

50 

75 

80 

50 

80 

85  88 

75 

80 

85 

50 

75 

80 

50 

80 

85  88 

75 

80 

85 

50 

75 

80 

50 

80 

85  88 

75 

80 

85 

50 

75 

80 

50 

80 

85  88 

Table  4-13  (Cont'd) 


Engage  (Cont'd) 

C.  Defense  0  0  0  0  0  0  50  50  60  50 

D.  Patrol  0000000000 

E.  Ammuni  _on  rationing  0  0  0  0  0  0  40  40  50  40 

G.  Concealment  0  0  0  0  0  0  25  25  40  25 

Team  fire  0  0  0  0  0  0  25  25  40  24 


I 

l 


| 
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Table  4-14 


Wwwwmm 


Frequency  Utility  Ratings  for  DORAC  Alternatives 
on  Training  Management 


r-i 

<o 

H 

V 

< 

, 

i-H 

co 

H 

H 

•w 

< 

i/i 

£ 

& 

cn 

w 

+ 

4* 

4* 

rH 

4* 

4* 

& 

/-*» 

/■“h 

S-'' 

/*“ N 

t-H 

CM 

<r 

f—i 

f— 1 

rH 

H 

f— 1 

»H 

+ 

'w' 

'w' 

v-' 

v-/ 

< 

VH 

v-< 

p* 

1 

u- 

P* 

b 

fn 

Pxh 

os 

Pi 

Pi 

pi 

V) 

Pi 

fxj 

OS 

Condition  to  operate 

A.  Calibrate  (zero) 


1. 

adjust  for  minute 
of  angle 

50 

75 

90 

0 

50 

50 

0 

50 

50 

80 

2. 

adjust  for  gravity 
effect 

50 

75 

90 

0 

50 

50 

0 

50 

50 

80 

3, 

sight  weapon 
alineraent 

50 

75 

90 

0 

50 

50 

0 

50 

50 

80 

B. 

Load 

1. 

inspect  top  round 
alinement 

75 

90 

98 

0 

75 

75 

75 

85 

98 

90 

2. 

apply  blow  to  seat 
magazine 

75 

90 

100 

100 

90 

90 

100 

90 

100 

100 

3. 

insure  the  first 
round  is  seated 
in  the  chamber 

75 

90 

100 

0 

75 

75 

75 

90 

100 

95 

Maintain  operation 

A. 

Reload 

1. 

remove  the 
magaz ine 

75 

90 

100 

0 

90 

90 

100 

90 

100 

100 

2. 

see  'load' 

B. 

Reduce  a  stoppage 

- 

— 

1. 

perform  immediate 
action  procedure 

35 

70 

90 

30 

40 

50 

35 

70 

75 

70 

2. 

inspect  an  M-16 
and  diagnose  the 
stoppage 

50 

75 

90 

0 

50 

50 

0 

50 

50 

8C 

3. 

take  the  appro- 
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Engage  (Cont'd) 
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For  example,  the  Effectiveness  score  of  "calibrate  (zero)"  for  alterna¬ 
tive  one,  RF,  on  Readiness  Evaluation  was  given  by: 

em  =*  (75)  (75) /100  -  56.25  (4-2) 

Effectiveness  ratings  for  the  DORAC  alternatives  on  the  two  WDs  are 
presented  in  Tables  4-15  and  4-16. 

Determine  Partial  Utilities  for  WDs.  Following  the  generation  of 
the  systems  effectiveness  array,  the  next  step  in  the  analysis  was  to 
produce  partial  utility  scores  for  each  alternative  on  each  WD.  The  gen¬ 
eral  form  of  the  equation  used  to  aggregate  effectiveness  ratings  across 
OMs  is  given  in  (4-3) : 

f[u(xij)]  =  l  ujk  eijk  (4'3) 

where  f[>]  is  the  partial  information  utility  score  of  the 
i  alternative  on  the  j  WD, 

u  .  is  the  utility  score  of  the  OH  nested  in 
J  the  jth  WD  (Tables  4-9  and  4-10), 

and  is  the  ef fectiveness^ating  of  the  i^  DORAC 

J  alternative  on  the  k  OM  nested  within 
the  j  WD  (Tables  4-15  and  4-16). 


Table  4-17  presents  the  partial  information  utility  (1U)  scores  of  each 
of  DORAC  alternatives  on  the  two  WDs,  Readiness  Evaluation  (RE)  and 
Training  Management  (TM) . 

Aggregate  Partial  Utility  Scores.  The  final  step  in  generating  the 
systems-versus-criteria  array  was  to  aggregate  partial  IU  scores  across 
WDs  to  produce  a  global  IU  score  for  each  alternative.  The  expression 
used  to  obtain  global  IU  scores  is  presented  as  follows: 


IUi  =  I  Wjf[u(xij)], 


(4-4) 


,th 


where  IU^  represents  the  global  IU  score  for  the  i 
DORAC  alternative, 

W,  is  the  importance  weight  for  the  j  WD  (page  4-27), 


j 


and  f[*]  is  the  partial  IU  score  of  the  iC^  alternative 
on  the  j  WD  (Table  4-17). 
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Table  4-16  (Cont’d) 
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Table  4-17 


Partial  Utility  Scores  for  DORAC  Alternatives 

Worth  Dimension 


Alternative 

RE 

TM 

1. 

RF  (1) 

29.25 

7.48 

2. 

RF  (2) 

30.90 

10.06 

3. 

RF(4) 

32.47 

12 . 07 

4. 

WP(1) 

7.71 

6.42 

5. 

RF (1)  +  WP(1) 

29.25 

10.50 

6. 

RF  (1)  +  WP  (3) 

30.40 

12.93 

7. 

SWAT(l) 

15.52 

15.72 

8. 

RF (1)  +  SWAT(l) 

37.58 

20.34 

9. 

RF (1)  +  SWAT (3) 

40.46 

34.08 

10. 

RF  +  WP  +  SWAT 

39.70 

26.05 

The  global  IU  scores  for  each  of  the  ten  DORAC  alternatives  used  in 
the  demonstration  CIEA  are  presented  in  Table  4-18. 


Table  4-18 

Information  Utility  Scores  for  DORAC  Alternatives 


Alternative 

IU  Score 

1. 

RF(1) 

25.23 

2. 

RF(2) 

27.05 

3. 

RF  (4) 

28.70 

4. 

WP(1) 

7.47 

5. 

RF'l)  +  WP(1) 

25.79 

6. 

RF(1)  +  WP (3) 

27.17 

7. 

SWAT(l) 

15.56 

8. 

RF (1)  +  SWAT(l) 

34,39 

9. 

RF (I)  +  SWAT (3) 

39.28 

10. 

RF  +  WP  +  SWAT 

37.18 
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Fct-im^ra  Costs  of  Alternatives.  The  next-to-the-last  step  in  defining 
the  systems-versus-criteria  array  involved  estimating  the  cost  of  each 
DORAC  alternative.  Cost  estimates  for  each  of  the  demonstration  DORAC 
alternatives  are  presented  in  Table  4-19.  The  cost  estimates  were  obtained 
usin  the  DORAC  Costing  Guidelines  presented  in  Hawley  and  Dawdy  (1981). 
Estimates  are  based  on  the  resources  required  to  accommodate  a  standard 
evaluation  unit  (i.e.,  Ft.  Bliss,  XX)  using  each  of  the  DORAC  alternatives. 
Additional  details  on  the  costing  process  are  provided  in  Appendix  D. 


Table  4-19 

Total  Estimated  Costs  for  DORAC  Alternatives 


Total  Estimated 

Alternativ  Cost  ($000* s) 


1. 

RF  (1) 

261.8 

2. 

RF(2) 

467.4 

3. 

RF  (4) 

846.0 

4. 

WP(1) 

79.2 

5. 

RF(1)  +  WP(1) 

341.7 

6. 

RF(1)  +  WP (3) 

438.2 

7. 

SWAT ( 1 ) 

106.2 

8. 

RF (1)  +  SWAT(l) 

382.9 

9. 

RF  (1)  +  SWAT  (3) 

498.9 

10. 

RF  +  WP  +  SWAT 

651.9 

Determine  Relative  Information  Worth.  Within  the  standard  context 
of  cost-effectiveness  analysis  (e.g.,  Barish  &  Kaplan,  1978;  TRADOC  Pam 
11-8) ,  Relative  Information  Worth  (RIW)  is  obtained  by  determining  Relative 
Information  Utility  fRIU)  and  Relative  Information  Cost  (RIC)  for  each 
alternative  and  then  combining  them.  RIU  for  alternative  i^  (RT.U^)  is 
determined  by  dividing  the  IU  measure  for  alternative  _i  (IU^)  by  that  of 
another,  usually  the  baseline  alternative: 

RIU±  =  IUi/lUb.  (4-5) 


3  The  cost  estimates  for  alternatives  involving  training  devices  are  based 
on  a  device  density  required  to  support  only  evaluation  activities. 


Relative  Information  Cost  for  alternative  1.  (RIC^)  is  determined  in  a 
similar  fashion: 


RICi  =  Ci/Cfa,  (4-6) 

t  h 

where  is  the  estimated  cost  of  the  i  alternative  and  is  the  esti¬ 
mated  cost  of  the  baseline  alternative. 

Applying  (4-5)  and  (4-6) ,  RIU  and  RIC  scores  for  the  demonstration 
DORAC  alternatives  were  obtained.  The  scores  are  presented  in  Table  4-20. 
Ranks  for  each  of  the  alternatives  are  also  provided. 


Table  4-20 

Relative  Information  Utility  and  Relative  Information  Cost 
for  DORAC  Alternatives 


Alternative 

RIU 

Rank 

RIC 

Rank 

1. 

RF(1) 

1.00 

8 

1.00 

3 

2. 

RF  (2) 

1.07 

6 

1.79 

7 

3. 

RF  (4) 

1.14 

4 

3.23 

10 

4. 

WP(1) 

0.30 

10 

0.30 

1 

5. 

RF  (1)  +  WP(1) 

1.02 

7 

1.31 

4 

6. 

RF(1)  +  WP(3) 

1.08 

5 

1.67 

6 

7. 

SWAT (1) 

0.62 

9 

0.41 

2 

8. 

RF (1)  +  SWAT (1) 

1.36 

3 

1.46 

5 

9. 

RF  (1)  +  SWAT  (3) 

1.56 

-L 

1.91 

8 

10. 

RF  +  WP  +  SWAT 

1.47 

2 

2.49 

9 

f  and 

RIC  are  descriptively  useful. 

but  have 

limited 

value  in  the 

of  an  analysis  like  CIEA 

where  the 

objective 

is  to 

identify  the 

most  cost-effective  DORAC  alternative.  To  determine  a  preferred  capa¬ 
bility, 

Worth 

was  defined  as  follows: 


RIU  and  RIC  were  integrated  into  a  measure  of  Relative  Information 
(RIW)  for  each  alternative.  The  RIW  of  the  i  alternative  (RIW^) 


RIW, 


RIU i/ RIC i 


I0i/Illb 

ci/cb 


(4-7) 
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An  RIW  score  greater  than  one  indicates  that  alternative  i_  is  more  cost 
and  information  effective  than  the  baseline  alternative.  What  is  done, 
in  effect,  is  to  normalize  system  cost  and  information  utility  relation¬ 
ships,  with  the  baseline  alternative  assigned  a  unit  value.  Table  4-21 
presents  the  RIW  scores  for  each  of  the  demonstration  DORAC  alternatives 


Table  4-21 

Relative  Information  Worth  Scores 
for  DORAC  Alternatives 


Alternative 

RIW 

Rank 

1. 

RF(1) 

1.00 

2.5  (tie) 

2. 

RF  (2) 

0.60 

8 

3. 

RF  (4) 

0.35 

30 

4. 

WP(1) 

1.00 

2.5  (tie) 

5. 

RF(1)  + 

WP(1) 

0.78 

6 

6. 

RF(1)  + 

WP(3) 

0.65 

7 

7. 

SWAT (1) 

1.51 

1 

8. 

RF (1)  + 

SWAT(l) 

0.93 

4 

9. 

RF (1)  + 

SWAT (3) 

0.82 

5 

10. 

RF  +  WP 

+  SWAT 

0.59 

9 

As  a  final  step  in  Phase  6,  all  of  the  system  evaluation  criteria 
were  assembled  into  a  formal  systems-versus-criteria  array.  In  this 
array,  the  rows  represent  DORAC  alternatives  and  the  columns  represent 
the  various  system  evaluation  criteria  (e.g.,  IU,  RIU,  IC,  RIC,  and  RIW) 
Table  4-22  presents  the  summary  systems-versus-criteria  array  for  the 
demonstration  DORAC  alternatives: 
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Table  4-22 


•*> 

Systems-Versus-Criteria  Array  for  DORAC  Alternatives 


Alternative 

IU 

RIU 

IC 

RIC 

RIW 

1. 

RF(l) 

25.23 

1.00 

261.8 

1.00 

1.00 

2. 

RF  (2) 

27.05 

1.07 

467.4 

1.79 

0.60 

3. 

RF(4) 

28.70 

1.14 

846.0 

3.23 

0.35 

4. 

WP(1) 

7.47 

0.30 

79.2 

0.30 

1.00 

5. 

RF (1)  +  WP(1) 

25.79 

1.02 

341.7 

1.31 

0.78 

6. 

RF (1)  +  WP(3) 

27.17 

1.08 

438.2 

1.67 

0.65 

7. 

SWAT(l) 

15.56 

0.62 

106.2 

0.41 

1.51 

8. 

RF(1)  +  SWAT (1) 

34.39 

1.36 

382.9 

1.46 

0.93 

9. 

RF (1)  +  SWAT ( 3 ) 

39.28 

1.56 

498.9 

1.91 

0.82 

10. 

RF  +  WP  +  SWAT 

37.18 

1.47 

651.9 

2.49 

0.59 

Perform  Sensitivity  Analyses 

It  is  likely  that  many  of  the  values  used  in  CIEA  to  specify  IU  or 
system  cost  wd.ll  be  based  upon  assumption,  expert  opinion,  or  incomplete  data 
and  thus  be  of  unknown  validity.  Sensitivity  analysis  refers  to  an  in¬ 
vestigation  of  the  effects  on  system  evaluation  criteria  of  estimated 
parameters  taking  on  values  different  from  those  used  in  the  analysis 
(Shannon,  1975).  Such  analyses  usually  consist  of  systematically  varying 
the  values  of  selected  decision  variables  over  a  range  of  interest  and 
observing  the  effects  of  these  changes  on  system  evaluation  criteria. 

Sensitivity  analysis  can  indicate  the  robustness  of  the  results  of 
a  CIEA.  It  is  desirable  to  determine  how  far  off  certain  parameter  esti¬ 
mates  can  be  without  changing  the  basic  conclusions  of  the  analysis.  If 
the  results  are  insensitive  to  a  fairly  wide  range  of  changes  in  selected 
parameters,  then  excessive  concern  need  not  be  given  to  the  accuracy  of 
these  parameters.  On  the  other  hand,  if  the  results  prove  to  be  highly 
dependent  on  the  values  of  certain  parameters,  it  might  be  prudent  to 
expend  additional  resources  and  obtain  more  precise  estimates  for  the 
parameters  in  question. 

In  CIEA,  one  candidate  for  sensitivity  investigation  is  the  effect 
of  estimated  system  cost  on  the  selection  of  a  preferred  alternative. 

The  first  step  in  the  conduct  of  a  cost  sensitivity  analysis  is  to  estab¬ 
lish  the  cost  range  over  which  the  top-rated  a’ternative  is  preferred; 
that  is,  to  determine  the  cost  estimate  that  would  make  the  top-rated 
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alternative  no  longer  preferred.  Then  determine  a  pessimistic  (i.e., 
highest)  cost  estimate  for  the  preferred  alternative.  The  pessimistic 
cost  estimate  can  be  obtained  by  reverting  to  the  cost  determination 
portion  of  the  CIEA  and  using  pessimistic  instead  of  expected  values  in 
the  cost  analysis.  Next,  compare  the  pessimistic  cost  estimate  with  the 
upper  bound  of  the  cost  range  where  the  top-rated  alternative  is  preferred. 
If  the  pessimistic  cost  estimate  is  below  the  upper  cost  bound,  the  top- 
rated  alternative  is  still  preferred.  However,  if  the  pessimistic  cost 
estimate  is  above  the  upper  cost  bound,  then  the  highest  rated  DORAC 
alternatives  should  be  examined  in  greater  detail  before  selecting  a 
preferred  capability. 

To  illustrate  the  cost  sensitivity  analysis  procedure,  consider  the 
top-rated  demonstration  DORAC  alternative  from  Table  4-22.  Alternative  7, 
SWAT  one  time  per  year,  has  an  RIU  score  of  0,62,  an  RIC  score  of  0.41, 
and  a  resulting  RIW  score  of  1.51.  The  second  rated  alternatives,  1  and 
4,  both  have  RIW  scores  of  1.00.  Following  the  procedure  outlined  in  the 
previous  paragraph,  the  RIC  value  for  alternative  7  would  have  to  exceed 
0.62  for  7's  RIW  score  to  be  less  than  the  scores  of  alternatives  1  and  4. 
That  is,  alternative  7's  pessimistic  cost  estimate  would  have  to  exceed 
$261,800  x  0.62  =  $162,316  before  the  results  of  the  analysis  would  be 
reconsidered  on  the  basis  of  cost  sensitivity  results  alone. 

Situations  may  also  arise  in  which  importance  ratings  for  WDs,  utility 
scores  for  OMs,  or  effectiveness  racings  for  DORIC  alternatives  are  at 
issue;  that  is,  varied  opinions  concerning  the  values  of  certain  of  these 
parameters  are  apparent  or  the  potential  impact  of  errors  or  mis judgments 
is  of  concern.  One  approach  to  resolving  issues  such  as  these  is  to  con¬ 
duct  a  sensitivity  investigation  generally  referred  to  as  parametric 
analysis.  In  a  parametric  analysis,  the  values  of  the  parameters  in  ques¬ 
tion  are  systematically  varied  over  a  range  of  interest  (e.g.,  the  range 
dictated  by  different  value  structures) .  The  effects  of  these  variations 
on  system  selection  criteria  are  observed.  If  the  parametric  analysis 
indicates  that  different  parameter  values  reflecting  different  points  of 
view  result  in  changes  in  system  rankings,  then  a  decision  must  be  made 
concerning  which  value  structure  is  to  dominate. 

In  the  present  demonstration  CIEA,  a  parametric  sensitivity  investi¬ 
gation  was  not  performed  because  no  radically  different  value  structures 
surfaced  during  the  parameter  elicitation  sessions. 


Select  Preferred  Alternative 


The  last  phase  in  CIEA  is  to  select  a  preferred  DORAC  alternative 
from  among  those  under  consideration.  In  many  situations,  the  decision 
rule  is  simple;  maximize  RIW.  This  choice  should  be  made  after  reviewing 
the  results  of  appropriate  sensitivity  analyses. 
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If  the  RIW  scores  for  a  number  of  alternatives  are  virtually  identi¬ 
cal,  an  appropriate  course  of  action  might  be  to  conclude  that  there  are 
no  differences  among  the  top-rated  alternatives.  In  such  situations,  the 
selection  of  a  preferred  DORAC  alternative  would  then  have  to  be  made  on 
the  basis  of  additional  analyses  or  on  the  basis  of  considered  military 
judgment. 

As  a  final  note,  a  special  selection  case  arises  when  one  or  more  of 
the  evaluation  criteria  (e.g.,  system  cost  or  information  utility)  is  sub¬ 
ject  to  a  constraint.  For  example,  it  will  often  be  the  case  that  there 
will  be  a  maximum  acceptable  cost  and  a  minimum  allowable  IU  value.  The 
rule  to  maximize  RIW  is  easily  adapted  to  situations  involving  maximum  or 
minimum  acceptable  values  for  a  given  evaluation  criterion.  When  such 
situations  arise,  simply  eliminate  alternatives  that  violate  constraints, 
regardless  of  their  overall  RIW  scores. 

Using  RIW  alone  as  a  basis  for  a  decision  concerning  a  preferred  al¬ 
ternative  should  be  done  cautiously.  The  results  of  the  demonstration 
CIEA  illustrate  exactly  why:  The  toprrated  alternative,  SWAT  one  time 
per  year,  has  an  IU  score  of  15.56  and  a  cost  of  $106,200.  It  might  be, 
for  example,  that  this  IU  score  would  be  judged  too  low.  If  that  were 
the  case,  alternative  7  (and  all  rating  below  it  on  IU)  should  be  dropped 
from  consideration.  The  result  of  such  a  decision  would  be  to  designate 
alternative  1,  RF,  as  the  preferred  alternative.  In  any  event,  the 
decision-making  group  should  review  both  the  RIU  and-  IC  results  carefully 
and  reject  alternatives  that  exceed  cost  constraints  or  do  not  provide  a 
sufficiently  high  value  for  IU. 


Discussion 


This  section  of  the  report  has  presented  the  results  of  a  demonstra¬ 
tion  application  of  a  preliminary  methodology  for  the  conduct  of  CIEA. 

The  results  of  the  application  indicate  that  the  preliminary  methodology 
(MAUM-CIEA)  is  usable.  Also,  the  DORAC  implementation  considered  in  the 
demonstration  exercise  (i.e.,  for  the  M16A1)  was  judged  to  have  potential 
utility  (as  indexed  by  the  IU  scores  of  the  alternatives). 

A  detailed  discussion  of  the  methodological  issues  underlying  the 
application  of  MAUM-CIEA  is  presented  in  Hawley  and  Dawdy  (1981).  To 
summarize  these  results,  the  following  points  are  noted: 

1.  MAUM-CIEA  is  valid  if  the  assumption  that  IU  can  be 
used  as  a  proxy  measure  for  strict  IW  is  found  to  be 
tenable. 
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2.  Given  point  1,  DORAC  evaluation  using  R1U  and  RIW 
is  valid  if  the  level  of  measurement  for  RIU  is  at 
least  equal- interval. 

The  objective  of  this  section  is,  however,  to  assess  the  feasibility  of 
the  operational  proaedures  involved  in  applying  MAUM-CIEA.  During  post¬ 
session  discussions  with  the  participants,  most  stated  that  they  did  not 
have  a  great  deal  of  difficulty  in  making  the  required  judgments.  Several 
points  relevant  to  improving  the  MAUM-CIEA  elicitation  process  were  noted. 
These  points  are  listed  as  follows: 

1.  Training  Management  per  se  represented  too  broad  a  range 
of  uses  to  be  included  as  a  single  WD.  TM  should  have 
been  separated  into  smaller  worth  units,  such  as  Basic 
Combat  Training,  Advanced  Individual  Training,  Resident 
School,  or  Unit  Training  Management. 

2.  With  respect  to  both  Training  Management  and  Readiness 
Evaluation,  explicit  provisions  should  be  made  for  the 
consideration  of  factors  such  as  personnel  turnover, 
personnel  aptitude  levels,  skill  decay  rates,  and  so 
forth.  Issues  such  as  these  impact  upon  the  utility  of 
different  types  of  information  for  different  uses  and 
the  frequency  with  which  information  should  be  received. 

3.  The  current  manual  MAUM  elicitation  procedure  places  an 
excessive  load  on  users.  The  elicitation  session  leader 
must  be  trained  in  the  use  of  the  methodology;  also, 
elicitation  sessions  are  time-consuming  and  tedious. 

These  points  suggest  that  the  acceptability  of  MAUM-CIEA 
would  be  greatly  enhanced  if  the  elicitation  procedure 
were  computer-aided. 
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SECTION  5 

SUMMARY  AND  RECOMMENDATIONS 
Summary 


The  objective  of  this  report.  was  to  present  the  results  of  an  analytical 
and  empirical  investigation  of  DORAC  feasibility  and  utility.  In  developing 
the  material,  three  separate  issues  impacting  upon  DORAC  feasibility  and 
utility  were  examined: 

1.  Acceptability 

2.  Technical  Feasibility 

3.  Financial  Feasibility 

Important  summary  points  relevant  to  each  of  these  issues  are  presented 
as  follows: 


Acceptability 

1.  The  Army’s  external  training  evaluation  system  (i.e.,  SQTs 
and  ARTEPs)  functions  much  as  formally  prescribed.  Internal 
evaluation  activities  are,  however,  characterized  as  deficient. 
The  primary  areas  of  deficiency  are  a  lack  of:  (1)  standard¬ 
ization  of  evaluation  procedures,  and  (2)  objective  per¬ 
formance  standards. 

2.  Field  personnel  suggested  a  number  of  revisions  to  improve 
the  TE  system.  Among  the  more  frequently  heard  suggestions 
were  the  following: 

More  realism  in  evaluation. 

Integrated  evaluation  procedures. 

Standardized  evaluation  procedures. 

Adherence  to  objective  performance  standards. 

3.  The  expressed  need  for  standardized  evaluation  procedures  is 
exactly  what  DORAC  provides.  DORAC  is  intended  to  support 
the  need  for  integrated  evaluation  procedures.  Also,  DORAC 

is  not  antithetical  to  the  need  for  more  realism  in  evaluation. 

4.  The  idea  of  using  training  devices  as  a  vehicle  for  the 
conduct  of  evaluation  activities  was  not  well  received. 

Field  personnel  generally  have  a  low  opinion  of  the  utility  of 
training  devices;  this  position  resulted  in  the  attitudes 
expressed.  The  results  from  ADA  survey  respondents  and 

the  reaction  of  the  M16A1  CIEA  participants  indicated,  how¬ 
ever,  that  the  use  of  training  devices  in  evaluation  can  be 
made  acceptable  to  field  personnel  if  the  concept  is  developed 
carefully. 


5.  In  terms  of  DORAC  concept  acceptability,  the  issue  was 

answered  with  a  qualified  "yes".  Given  appropriate  planning 
and  proper  support,  the  DORAC  concept  can  be  sold  to  field 
personnel.  The  keyword  is  sold,  however;  DORAC  will  not  be 
accepted  without  some  amount  of  resistance. 


Technical  Feasibility 

The  DORAC  concept  is  technically  feasible.  A  large  number  of  train¬ 
ing  devices  having  proficiency  assessment  potential  (although  not  ex¬ 
plicit)  currently  exist.  The  assessment  potential  of  projected  training 
devices  appears  even  more  promising.  The.  implementation  of  any  DORAC 
will,  however,  require  development  of  standardized  evaluation  procedures 
and  performance  standards  prior  to  fielding.  The  development  effort  will 
require  both  analysis  and  empirical  study.  Also,  for  the  development  of  a 
useful  capability,  a  procedure  for  identifying  the  most  critical  job 
activities  for  assessment  must  be  employed  (i.e.,  CIEA) .  The  apparent 
usability  of  the  preliminary  CIEA  methodology  is  encouraging  in  this 
respect. 


Financial  Feasibility 

If  a  DORAC  is  developed,  it  must  be  supported  to  a  greater  extent 
than  is  reportedly  true  for  current  training  devices.  It  should  be  recog¬ 
nized  that  the  resource  committment  required  to  effect  the  required  level 
of  support  is  potentially  quite  high.  This  point  again  underscores  the 
necessity  for  a  usable  methodology  for  identifying  the  most  cost-effective 
capability. 


Recommendations  for  Maximum  Payoff 


An  obvious  question  that  arises  from  all  of  the  previous  discussion 
is:  "Under  what  conditions  is  the  maximum  payoff  from  the  implementation 
of  the  DORAC  concept  likely  to  occur?"  Several  comments  relevant  to  this 
issue  are  presented  as  follows: 

1.  The  DORAC  concept  should  be  implemented  incrementally.  To 
overcome  potential  field  resistance  to  the  use  of  training 
devices  in  evaluation,  the  initial  application  should  be 
carefully  selected  so  as  to  maximize  the  likelihood  of  a 
visible  success.  Additional  applications  should  also  be 
selected  deliberately,  always  remembering  that  field 
personnel  must  accept  the  concept  for  it  to  be  successfully 
implemented. 
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2.  DORACs  should  be  used  to  provide  detailed  data  .'(down  to  the 
task  element  level)  on  performances  inherent  in  materiel 
system  use.  End-result  information  (i.e.,  summary  scores) 
should  also  be  provided.  In  its  initial  form,  the  DORAC 
concept  might  not  prove  useful  in  assessing  tactics,  doctrinal 
adherence,  or  other  more  global  performances. 

3.  When  possible,  DORAC  systems  should  be  designed  to  assess 
crew  proficiency,  as  well  as  individual  capabilities.  Field 
personnel  expressed  a  desire  for  integrated  performance 
evaluation.  Also,  the  successful  application  of  the  Air 
Defense  system  of  quasi-DORACs  (e.g.,  the  AN/MPQ-T1  Nike 
Hercules  simulator  and  the  AN/TPQ-29  IHAWK  simulator)  speaks 
well  for  this  point. 

4.  When  fielded,  DORAC  systems  should  be  designed  to  assess  a 
unit's  proficiency  on  its  primary  weapons  system(s) .  For 
example,  a  small  arms  DORAC  (e.g.,  one  based  on  SWAT  or 
Weaponeer)  would  likely  be  ill-received  in  an  Armored  unit, 
since  a  tank  unit  is  authi  ized  few  small  arms  and  does  not 
perceive  small  arms  proficiency  as  mission  essential. 

5.  When  possible,  DORACs  should  be  designed  to  be  integrated 
into  actual  materiel  system  equipment,  as  in  the  Air  Defense 
system  of  quasi-DORACs.  This  situation  would  tend  to  lessen 
the  overall  cost  of  acquiring  a  capability  and  also  assist 
in  overcoming  user  objections  to  the  DORAC  concept.  DORAC 
then  would  not  be  so  heavily  identified  with  training  devices. 

6.  DORAC  obtained  proficiency  information  should  be  included  in 
the  USR  much  as  the  personnel  and  equipment  data  are  now. 

That  is,  guidelines  for  assigning  the  training  rating  should 
be  provided.  Current  Air  Force  and  Army  aviation  readiness 
resporting  procedures  follow  a  similar  system.  When  tenta¬ 
tive  USR  modifications  have  been  established,  they  should 
not  be  implemented  all  at  once.  Rather,  changes  should  be 
phased  in  thus  allowing  commanders  time  to  establish  what 
their  actual  proficiency  levels  are,  and  to  use  the  informa¬ 
tion  to  re-orient  their  training  programs  in  order  to  reach 
desired  proficiency  levels.  In  this  manner,  the  DORAC  might 
be  perceived  initially  as  a  constructive  aide  rather  than 

as  a  purely  evaluation  tool  imposed  on  units  from  the  out¬ 
side. 
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Developmental  Recommendations 


The  previous  results  are  suggestive  of  a  series  of  recommendations 
concerning  the  next  developmental  steps  for  the  DORAC  concept.  These 
recommended  steps  are  listed  as  follows: 

1.  Develop  an  initial  DORAC  test  bed  for  training  evaluation 
and  proficiency  assessment.  Select  the  test  bed  where  an 
adequate,  accepted  training  device  is  available  and  in  use. 
Expend  the  effort  necessary  to  determine  the  most  cost- 
effective  DORAC;  standardize  evaluation  procedures;  and, 
where  required,  provide  realistic  performance  standards. 

2.  Monitor  and  evaluate  field  reaction  to  the  prototype  DORAC. 
Assess  what  changes  result  from  the  receipt  of  DORAC  data. 

3.  Introduce  a  modified  USR  reporting  procedure.  Assess 
user  reactions  to  its  use.  Evaluate  and  document  the 
effects  (if  any)  on  unit  functioning  of  improved  profi¬ 
ciency  reporting. 

4.  Integrate  all  initial  results  and  structure  a  trial  DORAC 
field  implementation  concept. 

5.  Select  an  independent  test  bed  unit  and  re-implement  DORAC 
with  a  minimum  of  external  interference.  Monitor  the 
situation  and  evaluate  the  results. 

6.  Make  a  decision  regarding  additional  implementation  of  the 
DORAC  concept.  At  this  point,  sufficient  empirical  data 
concerning  the  acceptability  of  the  concept  and  its  actual 
utility  should  be  available  to  enable  a  rational  decision 
to  be  made. 
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APPENDIX  A 


DORAC  INTERVIEW  GUIDE 


I.  What  kinds  of  formal  training  evaluation  do  you  now  do? 

1.  SQT: 

— external  (define) 

— internal  (define) 

2 .  ARTEP : 

— external 
— internal 

3.  Other 

How  are  training  evaluation  activities  (internal  and  external) 
scheduled? 

5.  Job  books? 

II,  How  frequently  do  you  conduct  training  evaluation? 

1.  SQT: 

— external 
— internal 

2.  ARTEP: 

— external 
— internal 

3.  Other: 


III,  Please  describe  your  procedures  for  the  conduct  of  each  of  the  training 
evaluations  that  you  do.  (Complete  the  following  for  SQT,  ARTEP,  etc.) 

1.  What  is  prescribed? 

2.  Where  do  you  obtain  the  required  materials? 

3.  Administrative  activities: 

— Who  is  responsible? 

— How  many  personnel  are  invol ved? 

— Where  is  it  conducted? 
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4.  Scoring  procedures: 

— How  established? 

— Who  evaluates? 

5.  Accountability: 

—To  higher  HQ? 

— Within  your  unit? 

6.  What  is  done  with  the  Information: 

— Within  your  unit? 

— Outside  your  unit? 


IV.  We  would  now  like  to  find  out  what  resources  (men,  materiel,  money) 
your  unit  uses  in  the  conduct  of  training  evaluation: 

1.  Preparation  time? 

2.  Training/preparation  for  administration,  scoring,  and  record  keeping? 

3.  POL/anrmunition/other  expendable  supplies? 

4.  Facilities  and  equipment? 

5 .  Personnel : 

— Evaluators? 

— Testees? 

6.  Transportation? 

7.  Record  keeping? 

8.  Coordination  with  other  organizations  (equipment,  evaluators,  etc.)? 

9.  Job  books? 


V.  In  your  opinion,  what  are  the  major  inadequacies  in  the  present 
training  evaluation  system? 

1.  Does  evaluation  result  in  skills  improvement? 

2.  Is  training  evaluation  supported  by  higher  HQ  in  terms  of  time, 
money,  and  materials? 

3.  Do  you  evaluate  the  skills  that  really  need  to  be  evaluated? 

4.  Are  the  results  a  valid  characterization  of  individual/unit 
combat  readiness? 

5.  Scheduling  problems? 
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VI.  In  what  ways  do  you  think  the  Army’s  training  evaluation  system  could 
be  improved  to  better  meet  its  intended  objectives— provide  feedback 
to  the  training  system,  characterize  individual/unit  proficiency? 


VII.  AR  220-1  states  that  unit  commanders  should  consider  SQT,  APTF.1%  and 
other  formal  training  evaluation  activities  when  formulating  the 
training  portion  of  the  Unit  Status  Report  (USR) .  How  do  you  determine 
your  unit's  training  readiness  level  for  input  into  the  USR? 


VIII.  It  has  been  suggested  that  one  method  of  improving  the  Army's  training 
evaluation  system  is  to  increase  the  use  of  trainers  (e.g.,  simulators, 
mock-ups,  etc.)  in  skills  evaluation. 

1.  Does  your  unit  currently  use  such  training  equipment  in  formal 
training  evaluation? 

2.  What  is  your  opinion  of  using  trainers  in  training  evaluation? 

3.  In  your  unit,  (where  additionally)  could  trainers  be  used  in 
evaluation?  (elaborate  on  potential) 

4.  What  about  the  inclusion  of  trainer-generated  skills  evaluation 
data  in  the  training  portion  of  your  USR?  (elaborate  on  potential 
and  attitude) 


APPENDIX  R 

DORAC  QUESTIONNAIRE 
STAFF  LEVEL 

Please  answer  the  following  questions.  Put  additional  comments  on  the 
back  of  the  page. 

1.  What  kinds  of  formal  training  evaluation  do  you  do  now? 

Type  of  evaluation  Yes  or  No  Frequency 


Kow  are  formal  training  evaluation  activities  scheduled? 


SQT: 


ARTEP : 


ORE: 


FIX: 


CPX: 


EDRE: 


FIRING  ARTEP: 


OTHER: 


3.  (Continued) 

No.  of 

Type  of  Personnel 

Evaluation  T  volved; 

SQT 

ARTEP 


ORE 


FIX 


CPX 


EDRE 


FIRING 

ARTEP 


ASP 


Other 

(list) 


3.  (Continued) 


Type  of 
Evaluation 


Standards 
Set  By: 


Accountable 
to : 


Evaluation 
Information 
Used  to: 


4,  What  resources  does  your  unit  use  in  the  conduct  of  formal  training 
evaluation? 

1  I  Training/Preparation  I 


Type  of 
Evaluation 


,  Preparation 
!  Time 


for  administration, 
scoring,  record 
keeping 


POL,  ammunition, 
other  expendables 


4 •  (Continued) 


Type  of 
Evaluation 


Facilities 


SQT 


ARTEP 


ORE 


FIX 


CPX 


EDRE 


FIRING 
A'  TEP 


ASP 


Other 

(List) 


Equipment 


I  Evaluators 


4.  (Continued) 


Type  of 

. 

Evaluation 

Testers 

--  - - -  1 

Transportat ion 

Record  keeping 

6.  What  kinds  of  internal  (i.e.,  intraunit)  training  evaluation  do  you 
do  now? 


Type  of  Evaluation 

Yes  or  No 

Frequency 

3QT 

ARTEP 

ORE 

FIX 

CPX 

EDRE 

Other  (list)  : 


8.  Complete  ehe  following  table  for  internal  (i.e.,  Intraunit)  training 
evaluations  (leave  non-applicable  squares  blank) 


Type  of 

Evaluat ion 

What  is 

Prescribed? 

Req'd  materials 
obtained  from: 

Responsible 

Agency 
(S3,  etc.) : 

SQT 

ARTEP 

ORE 

1 

- 

FTX 

L 

CPX 

EDRE 

Other 

(list) 


8,  (Continued) 

No .  of 

Type  of  Personnel 

Evaluat ion  Invo 1 ved : 

1  Location: 

1  Evaluator 

8.  (Continued) 


Evaluation 

Type  of 

Standards 

Accountable 

Information 

Evaluation 

Set  By: 

to: 

Used  to: 

9.  What  resources  does  your  unit  use  in  the  conduct  of  internal  (i.e.. 


intraunit)  training  evaluation? 


Type  of 
Evaluation 


reparation 


;  Training/Preparation 
for  administration, 
scoring,  record 
keeping 


POL,  ammunition, 
other  expendables 


9.  (Continued) 

i 


Type  of 
Evaluation 


Facilities 


Equipment 


Evaluators 


10.  Indicate  how  informal  training  evaluation  is  supported  by  higher  HQ. 

Time  Money  Materials  Personnel  Other 


\1.  For  what  HOSs  are  sufficient  job  books  not  available? 


12.  List  those  training  evaluations  which  you  feel  result  in  skills 

improvement.  Indicate  whether  the  evaluation  is  formal  or  internal. 


13.  Do  you  presently  evaluate  the  skills  that  really  need  to  be  evaluated? 


14.  Which  evaluation  results  yield  a  valid  characterization  of  individual/ 
unit  combat  readiness?  Why? 


15.  What  kinds  of  scheduling  pro'  ms  do  you  encounter? 


16.  Primary  objectives  of  the  Army’s  training  evaluation  system  are  to 

provide  feedback  to  the  training  system  and  to  characterize  individual/ 
unit  prof i-iency.  Briefly,  how  could  the  Army’s  training  evaluation 
system  be  improved  to  better  meet  these  objectives? 


17.  How  do  you  determine  your  unit’s  readiness  level  for  input  into  the  USR? 


18,  It  lias  been  suggested  that  one  method  of  improving  the  Army  s  training 
evaluation  system  is  to  increase  the  use  of  trainer  (e.g.,  simulators, 
mockups ,  etc,)  in  skills  evaluation.  If  your  unit  uses  such  devices 
in  evaluation,  fill  out  the  following  matrix. 


Type  of 

Trainer/ 

Simulator 


Type  of 
Evaluation 


Formal  or 

Informal 

Evaluation 


Availability 


19.  Briefly,  what  is  your  opinion  of  using  trainers  or  simulators  in 
training  evaluation? 


20.  In  your  unit,  where  could  trainers  or  simulators  be  used  in  evaluation? 
Elaborate  on  the  potential  of  using  such  devices. 


21.  For  the  craining  portion  of  your  USR,  could  you  use  skills  evaluation 
data  which  has  been  generated  by  trainers/simulators?  How  could  you 
use  such  information,  and  what  is  your  attitude  concerning  its  use? 


APPENDIX  C 


DORAC  QUESTIONNAIRE 
COMPANY/ BATTERY/TROOP 

Please  answer  the  following  questions.  Put  additional  comments  on  the 
back  of  the  page. 

1.  What  kinds  of  internal  (i.e.,  intraunit)  training  evaluation  do  you 
do  now? 


Other  (list) 


2.  How  are  internal  (i.e,  ,  intraunit)  training  evaluation  activities 
scheduled? 

SQT: 


ARTEP: 


ORE: 


FIX: 


CPX: 


EDRE: 


Other : 


C-2 


3.  Complete  the  following  table  for  internal  (i.e.  ,  intraunit)  training 
evaluations  (leave  non-applicable  squares  blank) : 

Responsible 

Type  of  What  is  Req'd  materials  Agency 

Evaluation  Prescribed?  obtained  from:  (S3,  etc.): 


3.  (Continued) 


Type  of 
Evaluat ion 


No.  of 
Personnel 

Involved:  Location 


3.  (Continued) 


Type  of 
Evaluation 


Standards 
Set  By: 


Accountable 
to : 


Evaluation 
Information 
Used  to: 


4.  What  resources  does  your  unit  use  in  the  conduct  of  internal  (i.e., 
intraunit)  training  evaluation? 

Training/ Preparation 
for  administration, 

Type  of  Preparation  scoring,  record  POL,  ammunition, 

Evaluation  Time  keeping  other  expendables 


4.  (Continued) 


Type  of 

Evaluation  Facilities  Equipment  Evaluators 


SQT 


ARTEP 


ORE 


FTX 


CPX 


DRE 


Other 

(List) 


4.  (Continued) 


Type  of 

Evaluation  Testers  Transportation  Record  keeping 


4.  (Continued) 

Coordination  with 

Type  of  other  units  (eqpt., 

Evaluation  evaluators,  etc.).  Job  books  Other 


rmal  training  evaluation  is  supported  by  higher  HQ. 


Money 


Materials 


Personnel 


Other 


6.  For  what  MOSs  are  sufficient  job  books  not  available? 


7.  List  those  training  evaluations  which  you  feel  result  in  skills 

improvement.  Indicate  whether  the  evaluation  is  formal  or  internal. 


8.  Do  you  presently  evaluate  the  skills  that  really  need  to  be  evaluated? 


9.  Which  evaluation  results  yield  a  valid  characterization  of  individual/ 
unit  combat  readiness?  Why? 


10.  What  kinds  of  scheduling  problems  do  you  encounter? 


C-ll 


11.  Primary  objectives  of  the  Army's  training  evaluation  system  are  to 

provide  feedback  to  the  training  system  and  to  characterize  individual/ 
unit  proficiency.  Briefly,  how  could  the  Army's  training  evaluation 
system  be  improved  to  better  meet  these  objectives? 


12.  It  has  been  suggested  that  one  method  of  improving  the  Army's  training 
evaluation  system  is  to  increase  the  use  of  trainer  (e.g.,  simulators, 
mockups ,  etc.)  in  skills  evaluation.  If  your  unit  uses  such  devices 
in  evaluation,  fill  out  the  following  matrix. 


Type  of 

Formal  or 

Trainer/ 

Type  of 

Informal 

Simulator 

Evaluation 

Evaluation 

Availability 

13.  Briefly,  what  is  your  opinion  of  using  trainers  or  simulators  in 
training  evaluation? 


14.  In  your  unit,  where  could  trainers  or  simulators  be  used  in  evaluation? 
Elaborate  on  the  potential  of  using  such  devices. 


15.  For  the  training  portion  of  your  USR,  could  you  use  skills  evaluation 
data  which  has  been  generated  by  trainers/simulators?  How  could  you 
use  such  information,  and  what  is  your  attitude  concerning  its  use? 


EDRE 


FIRING 

ARTEP 


17.  How  do  you  conduct  rifle  training? 


18.  Could  you  use  a  simulator  for  this,  if  one  was  available? 


APPENDIX  D 


Cost  Guide 
Cost  Matrix 

Number  of  Devices  Required 
Personnel  Costs 
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Note:  When  a  non-integer  results  from  a  calculation,  round  up. 


Acquisition  cost  cf  device,  (ACD)  Initial  Cost  of  Maintenance  Supplies  (ICMS) 
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Ex  =  Existing  $  =  given  in  1,000’s 

N/A  =  Not  available  Note:  See  D-2  &  D-3  for  explanation  for  tcp  margin  labels 

0JT  =  On-the-job  training 
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RF  2/Yr  2  1%  %  32.8  -  -  OJT  —  —  75  Ex  Ex  Ex  23.4  9.4 

WP  1/Yr  2  1/3  1/6  5.5  -  -  N/A  -  -  0  Ex  40  15  N/A  N/A 

SWAT  1/Yr  4  1%  h  14.5  -  -  N/A  —  -  0  Ex  125  20  N/A  N/A 


Number  of  Devices  Required 


Record  Fire:  Annual  Evaluation  Frequency  of  1 


if  Organizations  1 

BFL  “ 

74  X  4  * 

EFL  - 

296  X  1 

RE  - 

9%  X  296 

AFL  - 

27  +  296 

ECL  - 

1*4  hr. 

TBC  - 

*4  hr. 

SN  = 

2 

WSL  = 

2 

F.C/D  =  4X2 

1*5  =  ^ 

TDA  = 

343  X 
360 

DC  = 

228  X  4  = 

NDR  * 

323  * 

912 

74 
296 
296 
26.64 
=  323 


■vb-  --oafo**; 


r 


Number  of  Devices  Required 


Record  Fire:  Annual  Evaluation  Frequency  of  2 

AEF  =  2 

ECS  -  32 

OEUS  >-  32  _ 

E/0  -  120  -  X 

0E  =  120  -  3.75  - *  4 

32 

//  Organizations  =  74 


BFL 

74  X  4 

=  296 

EFL 

= 

296  X  2 

=  592 

RE 

x= 

9%  X  592 

=  53.28  - ► 

AFL 

= 

54  +  592 

=  646 

ECL 

= 

1*4  Hr. 

TBC 

= 

*5  Hr. 

SN 

*s 

2 

WSL 

= 

4 

EC/D 

= 

■  4X2 

=  8  =  4 

i*s  +  h 

2 

TDA 

3S 

343  X 
360 

240  =  228 

DC 

= 

228  X .4 

=  912 

NDR 

ae 

646  = 

.708  - *>  1 

912 


l 


D-7 


•Jr''* 


i 

\ 


<►**»•*• 


Number  of  Devices  Required 


Record  Fire:  Annual  Evaluation  Frequency  of  4 


AEF  =  4 

ECS  -  32 

OEUS  -  32 

E/0  -  120  -  X 

OE  =  120  =*  3.75  - -  4 

32 

#  Organizations  »<  74 

BFL  *  74  X  4  =  296 

EFL  =  296  X  4  =  1184 

RE  =  9%  X  1184  =  106.56  - 107 

AFL  =  107  +  1184  =  1291 

ECL  =  1*5  hr. 

TBC  =  h  hr. 

SN  *  2 

ViSL  =  4 

EC/O  =  4X2  =8=4 

1*5  +  *5  2 

TDA  =  343  X  240  =  228 

360 

DC  =  228  X  4  -  912 

NDR  =  1291  =  1.42  - *  2 

*912 
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Number  of  Devices  Required 


Weaponeer:  Annual  Evaluation  Frequency  of  1 


AEF 

=* 

1 

ECS 

- 

1 

OEUS 

= 

•  1 

E/0 

a 

120  *  X 

0E 

= 

120  >=  120 

1 

#  Organizations  = 

74 

BFL 

a 

74  X  120  - 

8880 

EFL 

a 

8880  X  1 

RE 

= 

5%  X  8380  • 

444 

AFL 

•«X 

444  +  8880 

=  9324 

ECL 

a 

1/3  hr. 

TBC 

- 

1/6  hr. 

SN 

= 

2 

WSL 

4 

EC/D 

■  4X2 

=  8  =  16 

1/3  +  1/6 

h 

TDA 

= 

365  X  240 
365 

=  240 

DC 

s 

240  X  16  - 

3840 

NDR 

= 

9324  =  2. 

428  - +  3 

3840 


3 


Number  of  Devices  Required 


Weaponeer:  Annual  Evaluation  Frequency  of  3 


AEF 

SS 

3 

ECS 

= 

1 

OEUS 

=1 

■  1 

E/0 

Xu 

120  -  X 

OE 

a 

120  *=  120 

1 

#  Organizations  = 

74 

BFL 

s: 

74  X  8880 

EFL 

= 

8880  X  3  = 

26640 

RE 

= 

5%  X  26640  = 

1332 

AFL 

1332  +  26640 

4-  27972 

ECL 

1/3  hr. 

TBC 

- 

1/6  hr. 

SN 

SS 

2 

WSL 

IS 

4 

EC/D 

= 

4X2 

=  8  = 

1/3  +  1/6 

h 

TDA 

= 

365  X  240 
365 

=  240 

DC 

me 

240  x  16  = 

3840 

NDR 

27072  =  7 

3840 

.284  - ► 
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Number  of  Devices  Required 


SWAT:  Annual  Evaluation  Frequency  of  3.  ( 

AEF  =  1 

ECS  -  5 

OEUS  -  4.5  _ 

E/0  -  120  -  X 

OE  «  120  «  27 

4.5 

if  Organizations  *74 
BFL  =  74  X  27  *  1998 

EFL  »  1998  X  1  *  1998 

RE  *  5%  X  1998  =  100 

AFL  =  100  +  1998  =  2098 

ECL  *  1*4  hr. 

TBC  *  k  hr. 

SN  *  4 

WSL  *  2 

EC/D  »  2X4  *  8  *  4 

Ik  +  k  2 


TDA  *  365  X  240  *  24G 

365 


Number  of  Devices  Required 


SWAT:  Annual  Evaluation  Frequency  of  3 

AEF  *  3 

ECS  *  5 

OEUS  "  4.5  _ 

E/0  -  120  =  X 

OE  =  120  =  27 

45 


#  Organizations  * 

74 

BFL  =  74  X  27  - 

1998 

EFL  =  1998  X  3  = 

5994 

RE  *  5%  X  5994  = 

299.7  - 

— -» 

AFL  =  300  +  5994 

=  6294 

ECL  =  1*4  hr. 

TBC  =  *4  hr. 

SN  =  4 

WSL  =  2 

EC/D  =  2X4 

8  =  4 

1*4  +  h 

2 

TDA  =  365  X  240 

-  240 

365 

DC  -  240  X  4  = 

960 

NDR  =  6294  =  6. 

556  - * 

7 

960 

D-12 


Direct  Personnel  Cost/Alternative 


Alternative  1 


Grade 

Number 

Cost 

Sub  -Total 

WG8 

1 

15,500* 

15,500 

E4 

3 

10,443 

31,329 

E6 

2 

14,562 

29,124 

E7 

1 

17,304 

17,304 

93, z57 

Alternative  2 

Grade 

Number 

Cost 

Sub-Total 

WG8 

1 

15,500* 

15,500 

E4 

5 

10,443 

52,215 

E6 

3 

14,562 

43,686 

E7 

1 

17,304 

17,304 

128,705 

Alternative  3 

Grade 

Number 

Cost 

Sub-Total 

WG8 

2 

15,500* 

31,000 

E4 

6 

10,443 

62,658 

E6 

4 

14,562 

58,248 

E7 

1 

17,504 

17,304 

169,210 

Alternative  4 

Grade 

Number 

Cost 

Sub-Total 

GS7 

.5 

15,317 

7,659 

E5 

3 

12,279 

36,837 

E7 

1 

17,304 

17,304 

61,800 


*  approximate 

Note:  Costs  from  TRADOC  cost  factors  handbook  effective  1  Oct  79. 
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Alternative  5 


Grade 


Number 


Cost 


Sub-Total 


WG8 

GS7 

E4 

E5 

E6 

E7 


1 

.5 

3 

3 

2 

2 


15,500* 

15,317 

10,443 

12,279 

14,562 

17,304 


15,500 

7,659 

31,329 

36,837 

29,124 

17,304 


155,057 


Alternative  6 

Grade 

GS7 

E4 

E5 

E6 

E7 

Alternative  7 

Grade 

GS7 

E5 

E7 


Number 

1 

3 

8 

3 

2 


Number 

,5 

3 

1 


Cost 

15,317 

10,443 

12,279 

14,562 

17,304 


Cost 

15,317 

12,279 

17,304 


Sub-Total 

15,317 

31,329 

98,232 

43,686 

34,608 

223,172 


Sub-Total 

7,659 

36,837 

17,304 

61,800 


Alternative  8 

Grade 

WG8 

GS7 

E4 

E5 

E6 

E7 


Number 

1 

,5 

3 

3 

3 

2 


uost 

15,500* 

15,317 

10,443 

12,279 

14,562 

17,304 


Sub-Total 

15,500 
7  659 
31,329 
36,837 
43,686 
34,608 

169,619 
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Alternative  9 


Grade 

Number 

Cost 

Sub-Total 

WG8 

1 

15,500* 

15,500 

GS7 

1 

15,317 

15,317 

E4 

3 

10,443 

31,329 

E5 

7 

12,279 

85,953 

E6 

3 

14,562 

43,686 

E7 

2 

17,304 

34,608 

226,393 

Alternative  10 

Grade 

Number 

Cost 

Sub-Total 

WG8 

1 

15,500 

15,500 

GS7 

1 

15,317 

15,317 

E4 

5 

10,443 

52,215 

E5 

6 

12,279 

73,674 

E6 

3 

14,562 

43,686 

E7 

3 

17,304 

51,912 

252,304 

D-15 


041383 


